更新redis的使用和数据汇总性能优化

This commit is contained in:
三千 2026-06-24 13:49:57 +08:00
parent 0b4860f46e
commit 4105b901f1
16 changed files with 662 additions and 119 deletions

23
.gitignore vendored
View File

@ -22,4 +22,25 @@ target
.idea .idea
# virtual machine crash logs, see http://www.java.com/en/download/help/error_hotspot.xml # virtual machine crash logs, see http://www.java.com/en/download/help/error_hotspot.xml
hs_err_pid* hs_err_pid*
# ============================================================
# Node.js / Frontend
# ============================================================
node_modules/
dist/
.DS_Store
*.local
# ============================================================
# Environment
# ============================================================
.env
.env.local
.env.*.local
# ============================================================
# Deploy
# ============================================================
weather-data-deploy/
weather-data-deploy.tar.gz

208
CLAUDE.md
View File

@ -2,53 +2,58 @@
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
> **See also** `AGENTS.md` for module breakdown, framework choices, conventions, and gotchas. This file supplements it with build commands and architectural patterns. > **See also** `AGENTS.md` for module breakdown, framework choices, conventions, and gotchas.
## Build & Run ---
```bash ## Project overview
# Full build (tests skipped by default per pom.xml <skipTests>true</skipTests>)
mvn clean install -DskipTests
# Build with tests **weather-data** is a full-stack meteorological data analysis platform: Java 17 / Spring Boot 3.5 multi-module backend + Vue 3 / Vite 5 / TypeScript frontend.
mvn clean install -DskipTests=false
# Run single test class
mvn -pl system-admin -DskipTests=false -Dtest=YourTestClass test
``` ```
weather-data/
**Run applications** from IntelliJ: ├── system-common/ → shared Java lib
- **Admin backend**: `com.weather.AdminApplication` (`system-admin/`) — port 8080, context path `/system-admin` ├── system-admin/ → admin backend (port 8080, /system-admin)
- **API service**: `com.weather.ApiApplication` (`system-api/`) — port 8081 ├── system-api/ → external API service (port 8081)
- **Code generator**: `com.weather.GeneratorApplication` (`renren-generator/`) ├── system-dynamic-datasource → multi-DS support (placeholder)
├── renren-generator/ → code generator
└── weather-data-ui/ → Vue 3 SPA frontend
```
**Database**: `weather_data_system` (MySQL). Init from `system-admin/db/mysql.sql`. Default admin: `admin` / `admin`. **Database**: `weather_data_system` (MySQL). Init from `system-admin/db/mysql.sql`. Default admin: `admin` / `admin`.
## Architecture ---
### Multi-module Maven project (Java 17, Spring Boot 3.5.x) ## Backend
### Build & Run
```bash
mvn clean install -DskipTests # full build (tests skipped by default)
mvn clean install -DskipTests=false # build with tests
mvn -pl system-admin -DskipTests=false -Dtest=YourTestClass test # single test
``` ```
weather-data (pom)
├── system-common → shared lib (all modules depend on this) Launch from IntelliJ:
├── system-admin → main admin backend - `AdminApplication` (`system-admin/`) → port 8080, context `/system-admin`
├── system-api → external API service - `ApiApplication` (`system-api/`) → port 8081
├── system-dynamic-datasource → multi-DS support (placeholder) - `GeneratorApplication` (`renren-generator/`)
└── renren-generator → code generator
```
### Service layer pattern ### Service layer pattern
All services extend one of two base classes from `system-common`: Two base classes in `system-common`:
- **`CrudService<Dao, Entity, DTO>`** — generic CRUD with `page()`, `get()`, `save()`, `update()`, `delete()`. The DTO type param is used for query criteria wrapping.
- **`BaseService<Dao>`** — lighter base without DTO generic.
New modules follow this convention: | Base | Purpose |
|---|---|
| `CrudService<Dao, Entity, DTO>` | Generic CRUD: `page()`, `get()`, `save()`, `update()`, `delete()` |
| `BaseService<Dao>` | Lighter base without DTO generic |
Module convention:
``` ```
modules/<name>/ modules/<name>/
├── controller/ → @RestController, returns Result ├── controller/ → @RestController, returns Result
├── dao/ → extends BaseMapper<Entity> (MyBatis-Plus) ├── dao/ → extends BaseMapper<Entity> (MyBatis-Plus)
├── dto/ → request/query DTOs (often extends BaseEntity for auto-fill) ├── dto/ → request/query DTOs (often extends BaseEntity)
├── entity/ → @TableName JPA entity ├── entity/ → @TableName JPA entity
├── service/ → interface extends CrudService/BaseService ├── service/ → interface extends CrudService/BaseService
│ └── impl/ → @Service implementation │ └── impl/ → @Service implementation
@ -60,48 +65,129 @@ Mapper XMLs: `src/main/resources/mapper/<domain>/**/*.xml`
### Key cross-cutting mechanisms ### Key cross-cutting mechanisms
| Mechanism | How it works | | Mechanism | How |
|---|---| |---|---|
| **Data permissions** | `@DataFilter` annotation on controller → `DataFilterAspect``DataFilterInterceptor` injects dept-based SQL filtering into MyBatis | | **Data permissions** | `@DataFilter` on controller → `DataFilterAspect` → MyBatis interceptor injects dept-based SQL |
| **Auto-fill** | `FieldMetaObjectHandler` fills `creator`/`createDate`/`updater`/`updateDate` via MyBatis-Plus meta-object handler | | **Auto-fill** | `FieldMetaObjectHandler` fills creator/date via MyBatis-Plus |
| **Scheduled jobs** | Quartz. Jobs in `schedule_job` table, implement `ITask`, annotated `@Component("beanName")`. `JobCommandLineRunner` auto-registers at startup | | **Scheduled jobs** | Quartz. `schedule_job` table, implements `ITask`, `@Component("beanName")` |
| **File scanning** | Three-part weather module: `WatchService` (primary, `FileWatchServiceManager`) + Quartz fallback (`FileScanTask`) + startup scan (`FileScanStartupRunner`). Files served via `FileDownloadController` | | **File scanning** | `WatchService` (primary) + Quartz fallback (`FileScanTask`) + startup runner |
| **Excel import** | EasyExcel with async progress tracking via `WeatherDataImportManager` | | **Excel import** | EasyExcel + async progress via `WeatherDataImportManager` |
| **API responses** | Always wrapped in `Result` class (`system-common`) | | **API responses** | Always wrapped in `Result` |
| **Validation** | Hibernate Validator on DTOs. XSS filter via `XssFilter` | | **Validation** | Hibernate Validator. XSS filter via `XssFilter` |
### PK strategy ### PK & Auth
`ASSIGN_ID` (Snowflake via `IdUtil.getSnowflakeNextId()`), set globally in MyBatis-Plus config. All entities extend `BaseEntity` which declares the `id` field. - PK: `ASSIGN_ID` (Snowflake). All entities extend `BaseEntity`.
- Auth: Apache Shiro 1.12 (Jakarta) + OAuth2 token. Login → `token` header.
- API module: `@Login` annotation + `AuthorizationInterceptor`.
### Auth flow ### Redis & Docs
- Apache Shiro 1.12 (Jakarta classifier) with OAuth2 token auth - Redis: optional, `project-options.redis.open` (default `false` in dev). `RedisAspect`.
- Login → get token → pass `token` header on subsequent requests - API docs: Knife4j at `/doc.html`, **disabled by default** (`knife4j.enable: false`).
- API module (`system-api`) uses `@Login` annotation + `AuthorizationInterceptor`
### Redis ### Weather domain (backend)
Optional, controlled by `project-options.redis.open` (default `false` in dev). Cache aspect: `RedisAspect`.
### API docs
Knife4j (Swagger UI) at `/doc.html`. **Disabled by default** (`knife4j.enable: false`). Enable only in dev profile.
## Custom Weather Domain
Three sub-modules under `system-admin/.../modules/weather/`: Three sub-modules under `system-admin/.../modules/weather/`:
| Sub-module | Purpose | Key detail | | Module | Purpose |
|---|---|---| |---|---|
| `dailydata/` | Daily weather observations | Excel batch import (async), EasyExcel listener pattern | | `dailydata/` | Daily observations, Excel batch import (async), EasyExcel listener |
| `station/` | Weather station CRUD | Linked to dept via `dept_id`, data-permission aware | | `station/` | Weather station CRUD, linked to dept via `dept_id` |
| `filescan/` | File monitoring & serving | WatchService → record → serve via `FileDownloadController` | | `filescan/` | File monitoring + serving. Format: `<地区>地区-<指标>.png` / `<地区>地区631信息.txt` |
File format convention (from `需求文档.md`): `<地区>地区-<指标>.png` for charts, `<地区>地区631信息.txt` for text data. `FileNameParser` extracts region/indicator keywords. Parameter `scan_root_path` in `sys_params` controls file-scan base directory.
## Database ---
Custom tables: `weather_daily_data`, `weather_station`, `weather_file_scan_record`. ## Frontend (`weather-data-ui/`)
Seed scripts in `system-admin/db/` (mysql.sql + Oracle/SQLServer/PostgreSQL/Dameng variants).
Parameter `scan_root_path` in `sys_params` controls the file-scan base directory. ### Commands
```bash
npm install # install dependencies
npm run dev # Vite dev server
npm run build / npm run build:prod # production build
npm run serve # preview production build
npm run lint # lint with autofix
npx vue-tsc --noEmit # type-check
```
### Stack
- Vite 5 + Vue 3 + TypeScript SPA
- Element Plus + Element Plus Icons for UI
- `vue-router` with hash history
- Pinia for state management
- Axios via `src/utils/http.ts` + `src/service/baseService.ts`
- API base URL: `VITE_APP_API` env var, overridable at runtime by `window.SITE_CONFIG.apiURL`
### Routing & state
- `src/router/base.ts`: base routes (`/`, `/home`, `/login`, etc.)
- `src/router/index.ts`: `beforeEach` guard — auth check, dynamic route registration from backend menus, tab management
- `src/store/index.ts` (`useAppStore`): user, permissions, dicts, dynamic routes, tabs
- `src/store/importTasks.ts`: long-running import task state for header indicator
- `src/utils/router.ts`: converts backend menu records → Vue router records, flattens nested routes for keep-alive
Layout is event-driven: `src/layout/` shell + `mitt` event bus (`src/utils/emits.ts`). Trace both the Pinia store and `mitt` events when changing navigation/sidebar/tabs/theme.
### Weather frontend module
The home dashboard (`src/views/home.vue`) uses a **composable-based architecture**. All domain logic is extracted from the SFC into `src/composables/`:
| Composable | Responsibility |
|---|---|
| `useWeatherConstants.ts` | Rain levels, temperature thresholds, filter field definitions, `fmtVal()`, level/class helpers |
| `useWeatherFilter.ts` | Filter state, toggle/reset/match logic, `matchOp()` |
| `useWeatherStats.ts` | `computeStats()`, `buildStatCards()`, `buildSummary()`, `rainLevelDistribution`, `WeatherDataRow` type |
| `useWeatherChart.ts` | ECharts dynamic import, `buildChartOption()`, `ResizeObserver`, precise trigger key (not deep watch) |
| `useWeatherExport.ts` | PNG/PDF export with dynamic `html2canvas`/`jspdf` imports, loading indicator |
Supporting utils:
- `src/utils/chartBuilder.ts` — chart option builders
- `src/utils/exportReport.ts` — shared `exportPNG()`/`exportPDF()`
### Critical rules learned (must follow)
#### 1. Null ≠ zero — missing data MUST be preserved as null
When mapping backend API responses to frontend models, **never** default missing numeric values to `0`. Rainfall of `0mm` means "no rain that day" (valid measurement); `null` means "no data available" (missing record). Use `: null` not `: 0` in data mapping, and display `"—"` for null values via `fmtVal()`.
```typescript
// ✅ Correct
rainfall: row.rain2020 != null ? +row.rain2020 : null,
// ❌ Wrong — confuses "no data" with "measured zero"
rainfall: row.rain2020 != null ? +row.rain2020 : 0,
```
All helper functions (`rainLevelLabel`, `tmaxValClass`, `tminValClass`, `rainValClass`) must accept `number | null` and return `"—"` or `""` for null. Stats computations (`computeStats`, `fStats`, `filterExtremes`) must skip null values in sums and extreme comparisons. ECharts will naturally render null as gaps in line/bar series.
#### 2. Heavy libraries must use dynamic imports
`html2canvas`, `jspdf`, and `echarts` are NOT imported at module level. They are loaded via `await import()` only when the user triggers export or chart rendering. This keeps them out of the initial bundle (~600KB saved).
#### 3. Google Fonts go in index.html, not scoped styles
Never use `@import url("https://fonts.googleapis.com/...")` inside Vue scoped styles — it blocks rendering. Instead, add `<link rel="preconnect">` + `<link rel="stylesheet">` in `index.html`.
#### 4. Export must show user feedback
When exporting images/PDFs, always show a loading indicator (`ElLoading.service` fullscreen) and a success/failure message (`ElMessage`). Disable the export button during rendering to prevent double-clicks.
#### 5. Deep watchers on filter objects are banned
Never use `watch(filters, callback, { deep: true })`. Instead, derive a precise computed trigger key that only includes fields actually affecting the output (e.g., `dataHash`, `filteredHash`, `extremesVersion`) and watch that.
### Common page pattern
Admin CRUD pages use `src/hooks/useView.ts` for shared list-page workflow: query, paging, sorting, delete, export, permission checks, dictionary lookup. Check whether behavior comes from `useView` before refactoring these screens.
### Other conventions
- Reusable selector/tree controls: `src/components/sys-*`, registered globally in `main.ts`
- SVG icons: `vite-plugin-svg-icons` from `src/assets/icons/svg/`
- Tests: no test runner configured yet
---
## Repository notes
- `README.md` does not contain substantive guidance; operational context lives in this file and `AGENTS.md`.
- `weather-data-ui/CLAUDE.md` is superseded by this merged file — the root `CLAUDE.md` covers both frontend and backend.

View File

@ -1,14 +1,204 @@
version: '2' # ============================================================
# Weather Data System - Docker Compose 部署编排
#
# 前置条件:
# 1. 安装 Docker Engine 20.10+ 和 Docker Compose v2
# 2. 复制 .env 文件并修改配置
# 3. 确保 JAR 文件已构建到对应 target/ 目录
# 4. 确保前端已构建到 weather-data-ui/dist/
#
# 启动: docker compose up -d
# 停止: docker compose down
# 日志: docker compose logs -f [服务名]
# ============================================================
services: services:
renren-admin:
image: renren/renren-admin # ==========================================================
ports: # MySQL 数据库
- "8080:8080" # ==========================================================
mysql:
image: mysql:8.0
container_name: weather-mysql
restart: unless-stopped
environment: environment:
- spring.profiles.active=dev MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD}
renren-api: MYSQL_DATABASE: ${MYSQL_DATABASE:-weather_data_system}
image: renren/renren-api MYSQL_USER: ${MYSQL_USER:-weather}
MYSQL_PASSWORD: ${MYSQL_PASSWORD}
TZ: Asia/Shanghai
ports: ports:
- "8081:8081" - "${MYSQL_PORT:-3306}:3306"
volumes:
- mysql-data:/var/lib/mysql
- ./deploy/mysql/init:/docker-entrypoint-initdb.d:ro # 初始化 SQL
command:
- --character-set-server=utf8mb4
- --collation-server=utf8mb4_unicode_ci
- --default-time-zone=+08:00
- --max-allowed-packet=128M
healthcheck:
test: ["CMD", "mysqladmin", "ping", "-h", "localhost", "-u", "root", "-p${MYSQL_ROOT_PASSWORD}"]
interval: 10s
timeout: 5s
retries: 10
start_period: 60s
networks:
- weather-net
# ==========================================================
# Redis 缓存
# ==========================================================
redis:
image: redis:7-alpine
container_name: weather-redis
restart: unless-stopped
command:
- redis-server
- --appendonly yes
- --requirepass ${REDIS_PASSWORD}
ports:
- "${REDIS_PORT:-6379}:6379"
volumes:
- redis-data:/data
healthcheck:
test: ["CMD", "redis-cli", "-a", "${REDIS_PASSWORD}", "ping"]
interval: 10s
timeout: 5s
retries: 5
networks:
- weather-net
# ==========================================================
# 管理后台 (Spring Boot Admin)
# ==========================================================
admin:
build:
context: ./system-admin
dockerfile: Dockerfile
image: weather-data-admin:${APP_VERSION:-latest}
container_name: weather-admin
restart: unless-stopped
depends_on:
mysql:
condition: service_healthy
redis:
condition: service_healthy
environment: environment:
- spring.profiles.active=dev SPRING_PROFILES_ACTIVE: ${SPRING_PROFILES_ACTIVE:-prod}
JAVA_OPTS: ${JAVA_OPTS:--Xms256m -Xmx512m -XX:+UseG1GC}
TZ: Asia/Shanghai
# 覆盖 application.yml 中的数据库/Redis 配置
SPRING_DATASOURCE_DRUID_URL: jdbc:mysql://mysql:3306/${MYSQL_DATABASE:-weather_data_system}?useUnicode=true&characterEncoding=UTF-8&serverTimezone=Asia/Shanghai&nullCatalogMeansCurrent=true&rewriteBatchedStatements=true
SPRING_DATASOURCE_DRUID_USERNAME: ${MYSQL_USER:-weather}
SPRING_DATASOURCE_DRUID_PASSWORD: ${MYSQL_PASSWORD}
SPRING_DATA_REDIS_HOST: redis
SPRING_DATA_REDIS_PORT: 6379
SPRING_DATA_REDIS_PASSWORD: ${REDIS_PASSWORD}
ports:
- "${ADMIN_PORT:-8080}:8080"
volumes:
- admin-logs:/app/logs
- admin-uploads:/app/upload
networks:
- weather-net
# ==========================================================
# API 服务
# ==========================================================
api:
build:
context: ./system-api
dockerfile: Dockerfile
image: weather-data-api:${APP_VERSION:-latest}
container_name: weather-api
restart: unless-stopped
depends_on:
mysql:
condition: service_healthy
redis:
condition: service_healthy
environment:
SPRING_PROFILES_ACTIVE: ${SPRING_PROFILES_ACTIVE:-prod}
JAVA_OPTS: ${JAVA_OPTS:--Xms256m -Xmx512m -XX:+UseG1GC}
TZ: Asia/Shanghai
SPRING_DATASOURCE_DRUID_URL: jdbc:mysql://mysql:3306/${MYSQL_DATABASE:-weather_data_system}?useUnicode=true&characterEncoding=UTF-8&serverTimezone=Asia/Shanghai&nullCatalogMeansCurrent=true&rewriteBatchedStatements=true
SPRING_DATASOURCE_DRUID_USERNAME: ${MYSQL_USER:-weather}
SPRING_DATASOURCE_DRUID_PASSWORD: ${MYSQL_PASSWORD}
SPRING_DATA_REDIS_HOST: redis
SPRING_DATA_REDIS_PORT: 6379
SPRING_DATA_REDIS_PASSWORD: ${REDIS_PASSWORD}
ports:
- "${API_PORT:-8081}:8081"
volumes:
- api-logs:/app/logs
- api-uploads:/app/upload
networks:
- weather-net
# ==========================================================
# 前端 UI
# ==========================================================
ui:
build:
context: ./weather-data-ui
dockerfile: ${UI_DOCKERFILE:-Dockerfile.offline}
image: weather-data-ui:${APP_VERSION:-latest}
container_name: weather-ui
restart: unless-stopped
depends_on:
- admin
- api
networks:
- weather-net
# ==========================================================
# Gateway - 统一入口 Nginx 反向代理
# ==========================================================
gateway:
image: nginx:alpine
container_name: weather-gateway
restart: unless-stopped
depends_on:
- ui
- admin
- api
ports:
- "${GATEWAY_PORT:-80}:80"
volumes:
- ./deploy/nginx/nginx.conf:/etc/nginx/conf.d/default.conf:ro
- gateway-logs:/var/log/nginx
healthcheck:
test: ["CMD", "wget", "-qO-", "http://localhost/"]
interval: 30s
timeout: 5s
retries: 3
networks:
- weather-net
# ============================================================
# 数据卷 (持久化存储)
# ============================================================
volumes:
mysql-data:
name: weather-mysql-data
redis-data:
name: weather-redis-data
admin-logs:
name: weather-admin-logs
admin-uploads:
name: weather-admin-uploads
api-logs:
name: weather-api-logs
api-uploads:
name: weather-api-uploads
gateway-logs:
name: weather-gateway-logs
# ============================================================
# 网络
# ============================================================
networks:
weather-net:
name: weather-net
driver: bridge

View File

@ -1,7 +1,23 @@
FROM java:8 FROM eclipse-temurin:17-jre-alpine
LABEL maintainer="weather-data"
LABEL description="Weather Data System - Admin Service"
RUN apk add --no-cache tzdata curl && \
cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \
echo "Asia/Shanghai" > /etc/timezone
WORKDIR /app
EXPOSE 8080 EXPOSE 8080
VOLUME /tmp ARG JAR_FILE=target/system-admin.jar
ADD renren-admin.jar /app.jar COPY ${JAR_FILE} /app/app.jar
RUN bash -c 'touch /app.jar'
ENTRYPOINT ["java","-jar","/app.jar"] ENV JAVA_OPTS="-Xms256m -Xmx512m -XX:+UseG1GC -Djava.security.egd=file:/dev/./urandom"
ENV SPRING_PROFILES_ACTIVE=prod
HEALTHCHECK --interval=30s --timeout=10s --retries=3 --start-period=60s \
CMD curl -f http://localhost:8080/system-admin/actuator/health || exit 1
ENTRYPOINT exec java ${JAVA_OPTS} -jar /app/app.jar

View File

@ -9,6 +9,8 @@ import com.weather.modules.weather.dailydata.vo.WeatherExcelVO;
import com.weather.modules.security.user.SecurityUser; import com.weather.modules.security.user.SecurityUser;
import com.weather.modules.security.user.UserContextHolder; import com.weather.modules.security.user.UserContextHolder;
import com.weather.modules.security.user.UserDetail; import com.weather.modules.security.user.UserDetail;
import com.weather.common.redis.RedisKeys;
import com.weather.common.redis.RedisUtils;
import lombok.AllArgsConstructor; import lombok.AllArgsConstructor;
import lombok.extern.slf4j.Slf4j; import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Component; import org.springframework.stereotype.Component;
@ -27,6 +29,7 @@ import java.util.concurrent.ConcurrentHashMap;
public class WeatherDataImportManager { public class WeatherDataImportManager {
private final WeatherDailyDataService weatherDailyDataService; private final WeatherDailyDataService weatherDailyDataService;
private final RedisUtils redisUtils;
private final Map<String, ImportProgress> tasks = new ConcurrentHashMap<>(); private final Map<String, ImportProgress> tasks = new ConcurrentHashMap<>();
public String submitImport(MultipartFile file) throws IOException { public String submitImport(MultipartFile file) throws IOException {
@ -82,6 +85,14 @@ public class WeatherDataImportManager {
progress.setStatus("COMPLETED"); progress.setStatus("COMPLETED");
log.info("导入任务 {} 完成,共导入 {} 行", taskId, progress.getProcessedRows().get()); log.info("导入任务 {} 完成,共导入 {} 行", taskId, progress.getProcessedRows().get());
// 导入完成后清除天气汇总缓存等待下次定时任务刷新
try {
redisUtils.deleteByPattern(RedisKeys.getWeatherSummarizePattern());
log.info("已清除天气汇总缓存");
} catch (Exception e) {
log.warn("清除天气汇总缓存失败", e);
}
} }
public ImportProgress getProgress(String taskId) { public ImportProgress getProgress(String taskId) {

View File

@ -1,5 +1,6 @@
package com.weather.modules.weather.dailydata.dao; package com.weather.modules.weather.dailydata.dao;
import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper;
import com.baomidou.mybatisplus.core.conditions.query.QueryWrapper; import com.baomidou.mybatisplus.core.conditions.query.QueryWrapper;
import com.baomidou.mybatisplus.core.toolkit.Constants; import com.baomidou.mybatisplus.core.toolkit.Constants;
import com.weather.common.dao.BaseDao; import com.weather.common.dao.BaseDao;
@ -18,7 +19,21 @@ import java.util.List;
@Mapper @Mapper
public interface WeatherDailyDataDao extends BaseDao<WeatherDailyDataEntity> { public interface WeatherDailyDataDao extends BaseDao<WeatherDailyDataEntity> {
List<WeatherDailyDataEntity> selectSummarizeList(@Param(Constants.WRAPPER) QueryWrapper<WeatherDailyDataEntity> wrapper); List<WeatherDailyDataEntity> selectSummarizeList(@Param(Constants.WRAPPER) LambdaQueryWrapper<WeatherDailyDataEntity> wrapper);
/**
* 按站点 + - + 年份范围查询汇总数据
* @param stationIds 站点 ID 列表
* @param month 月份 1-12
* @param day 1-31
* @param startYear 起始年份
* @param endYear 结束年份
*/
List<WeatherDailyDataEntity> selectSummarizeByMonthDay(@Param("stationIds") List<Long> stationIds,
@Param("month") int month,
@Param("day") int day,
@Param("startYear") int startYear,
@Param("endYear") int endYear);
int insertBatchMultiRow(@Param("list") List<WeatherDailyDataEntity> list); int insertBatchMultiRow(@Param("list") List<WeatherDailyDataEntity> list);
} }

View File

@ -3,11 +3,15 @@ package com.weather.modules.weather.dailydata.service.impl;
import cn.hutool.core.bean.BeanUtil; import cn.hutool.core.bean.BeanUtil;
import cn.hutool.core.collection.CollectionUtil; import cn.hutool.core.collection.CollectionUtil;
import cn.hutool.core.util.IdUtil; import cn.hutool.core.util.IdUtil;
import cn.hutool.core.util.ObjectUtil;
import cn.hutool.core.util.StrUtil; import cn.hutool.core.util.StrUtil;
import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper;
import com.baomidou.mybatisplus.core.conditions.query.QueryWrapper; import com.baomidou.mybatisplus.core.conditions.query.QueryWrapper;
import com.baomidou.mybatisplus.core.metadata.IPage; import com.baomidou.mybatisplus.core.metadata.IPage;
import com.weather.common.constant.Constant; import com.weather.common.constant.Constant;
import com.weather.common.page.PageData; import com.weather.common.page.PageData;
import com.weather.common.redis.RedisKeys;
import com.weather.common.redis.RedisUtils;
import com.weather.common.service.impl.CrudServiceImpl; import com.weather.common.service.impl.CrudServiceImpl;
import com.weather.common.utils.ConvertUtils; import com.weather.common.utils.ConvertUtils;
import com.weather.common.utils.TimeUtils; import com.weather.common.utils.TimeUtils;
@ -23,14 +27,18 @@ import com.weather.modules.weather.dailydata.service.WeatherDailyDataService;
import com.weather.modules.weather.station.dao.WeatherStationDao; import com.weather.modules.weather.station.dao.WeatherStationDao;
import com.weather.modules.weather.station.entity.WeatherStationEntity; import com.weather.modules.weather.station.entity.WeatherStationEntity;
import lombok.AllArgsConstructor; import lombok.AllArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.BeanUtils; import org.springframework.beans.BeanUtils;
import org.springframework.stereotype.Service; import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional; import org.springframework.transaction.annotation.Transactional;
import java.time.LocalDate; import java.time.LocalDate;
import java.time.ZoneId;
import java.util.*; import java.util.*;
import java.util.stream.Collectors; import java.util.stream.Collectors;
import java.util.stream.Stream;
@Slf4j
@Service @Service
@AllArgsConstructor @AllArgsConstructor
public class WeatherDailyDataServiceImpl extends CrudServiceImpl<WeatherDailyDataDao, WeatherDailyDataEntity, WeatherDailyDataDto> implements WeatherDailyDataService { public class WeatherDailyDataServiceImpl extends CrudServiceImpl<WeatherDailyDataDao, WeatherDailyDataEntity, WeatherDailyDataDto> implements WeatherDailyDataService {
@ -38,6 +46,7 @@ public class WeatherDailyDataServiceImpl extends CrudServiceImpl<WeatherDailyDat
private final SysDeptService sysDeptService; private final SysDeptService sysDeptService;
private final WeatherStationDao weatherStationDao; private final WeatherStationDao weatherStationDao;
private final WeatherDailyDataDao weatherDailyDataDao; private final WeatherDailyDataDao weatherDailyDataDao;
private final RedisUtils redisUtils;
@Override @Override
public PageData<WeatherDailyDataDto> page(Map<String, Object> params) { public PageData<WeatherDailyDataDto> page(Map<String, Object> params) {
@ -61,7 +70,7 @@ public class WeatherDailyDataServiceImpl extends CrudServiceImpl<WeatherDailyDat
private List<Long> getPrioritizedStationIds() { private List<Long> getPrioritizedStationIds() {
UserDetail user = SecurityUser.getUser(); UserDetail user = SecurityUser.getUser();
if (user == null || user.getDeptId() == null) return Collections.emptyList(); if (ObjectUtil.isEmpty(user) || user.getDeptId() == null) return Collections.emptyList();
List<Long> deptIds = sysDeptService.getSubDeptIdList(user.getDeptId()); List<Long> deptIds = sysDeptService.getSubDeptIdList(user.getDeptId());
if (deptIds.isEmpty()) return Collections.emptyList(); if (deptIds.isEmpty()) return Collections.emptyList();
List<WeatherStationEntity> stations = weatherStationDao.selectList( List<WeatherStationEntity> stations = weatherStationDao.selectList(
@ -70,8 +79,11 @@ public class WeatherDailyDataServiceImpl extends CrudServiceImpl<WeatherDailyDat
.in("dept_id", deptIds)); .in("dept_id", deptIds));
return stations.stream() return stations.stream()
.map(s -> { .map(s -> {
try { return Long.valueOf(s.getStationCode()); } try {
catch (NumberFormatException e) { return null; } return Long.valueOf(s.getStationCode());
} catch (NumberFormatException e) {
return null;
}
}) })
.filter(Objects::nonNull) .filter(Objects::nonNull)
.collect(Collectors.toList()); .collect(Collectors.toList());
@ -110,7 +122,7 @@ public class WeatherDailyDataServiceImpl extends CrudServiceImpl<WeatherDailyDat
.map(WeatherDailyDataEntity::getStationId) .map(WeatherDailyDataEntity::getStationId)
.filter(Objects::nonNull) .filter(Objects::nonNull)
.distinct() .distinct()
.collect(Collectors.toList()); .toList();
Map<Long, Long> stationDeptMap = new HashMap<>(); Map<Long, Long> stationDeptMap = new HashMap<>();
if (!stationIds.isEmpty()) { if (!stationIds.isEmpty()) {
List<String> stationCodeList = stationIds.stream().map(String::valueOf).collect(Collectors.toList()); List<String> stationCodeList = stationIds.stream().map(String::valueOf).collect(Collectors.toList());
@ -134,7 +146,8 @@ public class WeatherDailyDataServiceImpl extends CrudServiceImpl<WeatherDailyDat
entity.setUpdateDate(now); entity.setUpdateDate(now);
} }
weatherDailyDataDao.insertBatchMultiRow(list); int successCount = weatherDailyDataDao.insertBatchMultiRow(list);
log.info("成功插入{}条数据",successCount);
return true; return true;
} }
@ -169,11 +182,78 @@ public class WeatherDailyDataServiceImpl extends CrudServiceImpl<WeatherDailyDat
endTime = TimeUtils.convertToLocalDateTime(queryDto.getEndtDate()); endTime = TimeUtils.convertToLocalDateTime(queryDto.getEndtDate());
} }
QueryWrapper<WeatherDailyDataEntity> wrapper = new QueryWrapper<>(); // 尝试从缓存获取"历年同月同日"查询可命中缓存
wrapper.in("station_id", stationIds) List<DailyWeatherSummarizeDto> cached = getCachedSummarize(stationIds, startTime, endTime);
.between("observe_date", startTime, endTime); if (cached != null) {
return cached;
}
List<WeatherDailyDataEntity> list = weatherDailyDataDao.selectSummarizeList(wrapper); // 缓存未命中查询数据库
return querySummarizeFromDb(stationIds, startTime, endTime);
}
/**
* 尝试从 Redis 缓存获取汇总数据
*
* @return 缓存命中时返回数据未命中返回 null
*/
@SuppressWarnings("unchecked")
private List<DailyWeatherSummarizeDto> getCachedSummarize(
List<Long> stationIds, LocalDate start, LocalDate end) {
// 仅当月-日相同的跨年查询可命中缓存
if (start.getMonth() != end.getMonth() || start.getDayOfMonth() != end.getDayOfMonth()) {
return null;
}
String key = RedisKeys.getWeatherSummarizeKey(end.getMonthValue(), end.getDayOfMonth());
Object cached = redisUtils.get(key);
if (cached == null) {
return null;
}
Map<String, List<DailyWeatherSummarizeDto>> grouped;
try {
grouped = (Map<String, List<DailyWeatherSummarizeDto>>) cached;
} catch (ClassCastException e) {
return null;
}
// 验证所有请求站点都在缓存中任一缺失则降级到 DB
for (Long sid : stationIds) {
if (!grouped.containsKey(String.valueOf(sid))) {
return null;
}
}
// 从缓存中按站点过滤 + 限定年份范围Key String 避免 JSON 反序列化时 Long/Integer 类型丢失
return stationIds.stream()
.flatMap(sid -> {
List<DailyWeatherSummarizeDto> stationData = grouped.get(String.valueOf(sid));
return stationData != null ? stationData.stream() : Stream.empty();
})
.filter(d -> {
LocalDate obsDate = d.getObserveDate().toInstant()
.atZone(ZoneId.systemDefault())
.toLocalDate();
return !obsDate.isBefore(start) && !obsDate.isAfter(end);
})
.collect(Collectors.toList());
}
/**
* 直接查询数据库获取汇总数据
* <p> MONTH/DAY 定位具体日期 + YEAR BETWEEN 限定年份范围避免日期 BETWEEN 全量问题</p>
*/
private List<DailyWeatherSummarizeDto> querySummarizeFromDb(
List<Long> stationIds, LocalDate startTime, LocalDate endTime) {
List<WeatherDailyDataEntity> list = weatherDailyDataDao.selectSummarizeByMonthDay(
stationIds,
startTime.getMonthValue(),
startTime.getDayOfMonth(),
startTime.getYear(),
endTime.getYear());
return BeanUtil.copyToList(list, DailyWeatherSummarizeDto.class); return BeanUtil.copyToList(list, DailyWeatherSummarizeDto.class);
} }

View File

@ -0,0 +1,65 @@
package com.weather.modules.weather.dailydata.task;
import cn.hutool.core.bean.BeanUtil;
import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper;
import com.weather.common.redis.RedisKeys;
import com.weather.common.redis.RedisUtils;
import com.weather.modules.job.task.ITask;
import com.weather.modules.weather.dailydata.dao.WeatherDailyDataDao;
import com.weather.modules.weather.dailydata.dto.DailyWeatherSummarizeDto;
import com.weather.modules.weather.dailydata.entity.WeatherDailyDataEntity;
import jakarta.annotation.Resource;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Component;
import java.time.LocalDate;
import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;
/**
* 天气汇总缓存预热任务
* <p>
* 每日凌晨 1:00 执行将历年当月-当日的气象数据
* 按站点分组写入 Redis export-sum 接口查询
*
* @author 2333 123
* @since 1.0.0 2026-06-24
*/
@Slf4j
@Component("weatherSummarizeCacheTask")
public class WeatherSummarizeCacheTask implements ITask {
@Resource
private WeatherDailyDataDao weatherDailyDataDao;
@Resource
private RedisUtils redisUtils;
@Override
public void run(String params) {
LocalDate today = LocalDate.now();
int month = today.getMonthValue();
int day = today.getDayOfMonth();
log.info("开始刷新天气汇总缓存,日期: {}-{}", month, day);
try {
LambdaQueryWrapper<WeatherDailyDataEntity> wrapper = new LambdaQueryWrapper<>();
wrapper.apply("MONTH(observe_date) = {0} AND DAY(observe_date) = {1}", month, day);
List<WeatherDailyDataEntity> list = weatherDailyDataDao.selectSummarizeList(wrapper);
Map<String, List<DailyWeatherSummarizeDto>> grouped = list.stream()
.map(e -> BeanUtil.copyProperties(e, DailyWeatherSummarizeDto.class))
.collect(Collectors.groupingBy(dto -> String.valueOf(dto.getStationId())));
String key = RedisKeys.getWeatherSummarizeKey(month, day);
redisUtils.set(key, grouped, RedisUtils.NOT_EXPIRE);
log.info("天气汇总缓存刷新完成,共 {} 条记录,{} 个站点", list.size(), grouped.size());
} catch (Exception e) {
log.error("天气汇总缓存刷新失败", e);
}
}
}

View File

@ -7,7 +7,9 @@ import com.weather.modules.sys.entity.SysDeptEntity;
import com.weather.modules.sys.service.SysParamsService; import com.weather.modules.sys.service.SysParamsService;
import lombok.AllArgsConstructor; import lombok.AllArgsConstructor;
import lombok.extern.slf4j.Slf4j; import lombok.extern.slf4j.Slf4j;
import org.springframework.boot.CommandLineRunner; import org.springframework.boot.context.event.ApplicationReadyEvent;
import org.springframework.context.event.EventListener;
import org.springframework.scheduling.annotation.Async;
import org.springframework.stereotype.Component; import org.springframework.stereotype.Component;
import java.io.IOException; import java.io.IOException;
@ -15,18 +17,33 @@ import java.nio.file.Files;
import java.nio.file.Path; import java.nio.file.Path;
import java.nio.file.Paths; import java.nio.file.Paths;
import java.util.List; import java.util.List;
import java.util.concurrent.TimeUnit;
@Slf4j @Slf4j
@Component @Component
@AllArgsConstructor @AllArgsConstructor
public class FileScanStartupRunner implements CommandLineRunner { public class FileScanStartupRunner {
/**
* 启动后延迟扫描时间给应用留出充分的初始化时间
*/
private static final long STARTUP_DELAY_SECONDS = 30;
private final SysParamsService sysParamsService; private final SysParamsService sysParamsService;
private final SysDeptDao sysDeptDao; private final SysDeptDao sysDeptDao;
private final FileWatchServiceManager fileWatchServiceManager; private final FileWatchServiceManager fileWatchServiceManager;
@Override @Async
public void run(String... args) { @EventListener(ApplicationReadyEvent.class)
public void onApplicationReady() {
log.info("应用已启动,{} 秒后开始文件扫描目录初始化...", STARTUP_DELAY_SECONDS);
try {
TimeUnit.SECONDS.sleep(STARTUP_DELAY_SECONDS);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
log.warn("文件扫描延迟等待被中断,直接开始初始化");
}
String rootPath = sysParamsService.getValue(Constant.FILE_SCAN_ROOT_PATH); String rootPath = sysParamsService.getValue(Constant.FILE_SCAN_ROOT_PATH);
if (StrUtil.isBlank(rootPath)) { if (StrUtil.isBlank(rootPath)) {
log.warn("FILE_SCAN_ROOT_PATH 未配置,跳过文件扫描目录初始化"); log.warn("FILE_SCAN_ROOT_PATH 未配置,跳过文件扫描目录初始化");

View File

@ -6,26 +6,6 @@ spring:
url: jdbc:mysql://localhost:3306/weather_data_system?useUnicode=true&characterEncoding=UTF-8&serverTimezone=Asia/Shanghai&nullCatalogMeansCurrent=true&rewriteBatchedStatements=true url: jdbc:mysql://localhost:3306/weather_data_system?useUnicode=true&characterEncoding=UTF-8&serverTimezone=Asia/Shanghai&nullCatalogMeansCurrent=true&rewriteBatchedStatements=true
username: root username: root
password: root password: root
#达梦8
# driver-class-name: dm.jdbc.driver.DmDriver
# url: jdbc:dm://192.168.10.10:5236/renren_security?useUnicode=true&characterEncoding=UTF-8&serverTimezone=Asia/Shanghai&nullCatalogMeansCurrent=true
# username: renren_security
# password: 12345678
# #Oracle
# driver-class-name: oracle.jdbc.OracleDriver
# url: jdbc:oracle:thin:@192.168.10.10:1521:xe
# username: renren_security
# password: 123456
# #SQLServer
# driver-class-name: com.microsoft.sqlserver.jdbc.SQLServerDriver
# url: jdbc:sqlserver://localhost:1433;DatabaseName=renren_security
# username: sa
# password: 123456
# #postgresql
# driver-class-name: org.postgresql.Driver
# url: jdbc:postgresql://192.168.10.10:5432/postgres
# username: postgres
# password: 123456
initial-size: 10 initial-size: 10
max-active: 100 max-active: 100
min-idle: 10 min-idle: 10

View File

@ -40,7 +40,7 @@ spring:
data: data:
redis: redis:
database: 0 database: 0
host: 192.168.10.10 host: 127.0.0.1
port: 6379 port: 6379
password: # 密码(默认为空) password: # 密码(默认为空)
timeout: 6000ms # 连接超时时长(毫秒) timeout: 6000ms # 连接超时时长(毫秒)
@ -54,7 +54,7 @@ spring:
# 是否开启redis缓存 true开启 false关闭 # 是否开启redis缓存 true开启 false关闭
project-options: project-options:
redis: redis:
open: false open: true
#mybatis #mybatis
mybatis-plus: mybatis-plus:

View File

@ -42,6 +42,19 @@
ORDER BY station_id, observe_date ORDER BY station_id, observe_date
</select> </select>
<select id="selectSummarizeByMonthDay" resultMap="weatherDailyDataMap">
SELECT <include refid="summarizeColumns" />
FROM weather_daily_data
WHERE station_id IN
<foreach collection="stationIds" item="sid" open="(" separator="," close=")">
#{sid}
</foreach>
AND MONTH(observe_date) = #{month}
AND DAY(observe_date) = #{day}
AND YEAR(observe_date) BETWEEN #{startYear} AND #{endYear}
ORDER BY station_id, observe_date
</select>
<insert id="insertBatchMultiRow" parameterType="list" useGeneratedKeys="false"> <insert id="insertBatchMultiRow" parameterType="list" useGeneratedKeys="false">
INSERT INTO weather_daily_data ( INSERT INTO weather_daily_data (
id, station_id, observe_date, id, station_id, observe_date,

View File

@ -1,7 +1,23 @@
FROM java:8 FROM eclipse-temurin:17-jre-alpine
LABEL maintainer="weather-data"
LABEL description="Weather Data System - API Service"
RUN apk add --no-cache tzdata curl && \
cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \
echo "Asia/Shanghai" > /etc/timezone
WORKDIR /app
EXPOSE 8081 EXPOSE 8081
VOLUME /tmp ARG JAR_FILE=target/system-api.jar
ADD renren-api.jar /app.jar COPY ${JAR_FILE} /app/app.jar
RUN bash -c 'touch /app.jar'
ENTRYPOINT ["java","-jar","/app.jar"] ENV JAVA_OPTS="-Xms256m -Xmx512m -XX:+UseG1GC -Djava.security.egd=file:/dev/./urandom"
ENV SPRING_PROFILES_ACTIVE=prod
HEALTHCHECK --interval=30s --timeout=10s --retries=3 --start-period=60s \
CMD curl -f http://localhost:8081/renren-api/actuator/health || exit 1
ENTRYPOINT exec java ${JAVA_OPTS} -jar /app/app.jar

View File

@ -23,7 +23,7 @@ public class RedisAspect {
/** /**
* 是否开启redis缓存 true开启 false关闭 * 是否开启redis缓存 true开启 false关闭
*/ */
@Value("${weather.redis.open: false}") @Value("${project-options.redis.open: false}")
private boolean open; private boolean open;
@Around("execution(* com.weather.common.redis.RedisUtils.*(..))") @Around("execution(* com.weather.common.redis.RedisUtils.*(..))")

View File

@ -55,4 +55,18 @@ public class RedisKeys {
public static String getUserPermissionsKey(Long userId){ public static String getUserPermissionsKey(Long userId){
return "sys:user:permissions:" + userId; return "sys:user:permissions:" + userId;
} }
/**
* 天气汇总缓存Key按站点分组-日维度
*/
public static String getWeatherSummarizeKey(int month, int day) {
return "weather:summarize:" + month + ":" + day;
}
/**
* 天气汇总缓存匹配模式
*/
public static String getWeatherSummarizePattern() {
return "weather:summarize:*";
}
} }

View File

@ -116,4 +116,23 @@ public class RedisUtils {
public Object rightPop(String key) { public Object rightPop(String key) {
return redisTemplate.opsForList().rightPop(key); return redisTemplate.opsForList().rightPop(key);
} }
/**
* 按模式删除缓存
* @param pattern 匹配模式 "weather:summarize:*"
*/
public void deleteByPattern(String pattern) {
java.util.Set<String> keys = redisTemplate.keys(pattern);
if (keys != null && !keys.isEmpty()) {
redisTemplate.delete(keys);
}
}
/**
* 判断缓存是否存在
*/
public boolean exists(String key) {
Boolean result = redisTemplate.hasKey(key);
return Boolean.TRUE.equals(result);
}
} }