Integration and developing a new system to collect, store and analyze big data

Bayleaf developed a centralized data repository and analysis system to support data managements and the generation of catch estimates for Food, Social, and Ceremonial (FSC) and recreational fisheries within the pacific region.

Client Name

Fisheries and Oceans Canada

Application

Services

  • Project Management
  • Business Analysis
  • Graphic Design
  • Custom Web Development
  • Custom Desktop Development
  • Data Migration
  • Integration with numerous regional data systems

Technologies

  • .NET, MVC, Entity, Linq
  • ColdFusion
  • VB and VB.NET
  • XML
  • R, RODBC, RServe
  • Web services and WCF
  • Jasper Reporting Server
  • EXT, jQuery, DevExpress
  • Oracle, SQL Express, PL/SQL

Client Needs

Recreational and FSC harvest data is highly dispersed throughout the Pacific region. Data, stored and collected through multiple channels (custom applications, email, FTP, spreadsheets and paper file silos) were uploaded to a legacy desktop application with an Oracle database backend. Data was captured into the system using a combination of formal and ad hoc mechanisms (i.e.: transformative and migration scripts), and lacked consistency in collection, coding, accountability and overall management. Missing, unclear and inconsistent data made it difficult for staff to find and access data sets, much less produce accurate and insightful reports. The client needed: a centralized system to collect recreational and FSC harvest data, to standardize data quality, produce quantifiable catch estimates, provide effective business reporting and to enable data exchange with other regional and federal data systems.

Our Solutions

Project Management and Business Analysis

During our analysis process, we met with a variety of stakeholder groups to understand both their data collection process, methods of collection and analysis and reporting requirements. Results of the meetings included: defining a common naming system of data attributes, determining consistency in forms and data collection methods, and most importantly defining a standard as to how data was coded to reference data (e.g. gear, species, area, etc). Additionally, a solution architecture was defined that allowed for a more flexible storage approach of collected data. The service oriented architecture standardized integration between remote systems and the developed system. It also improved the analysis module by providing the flexibility different Fisheries operating within the region needed.

Systems Integration

The development of the KREST user interface (UI) focused on data management, analysis, reporting and administration. It integrates with Fisheries and Oceans Jasper Reporting Server and Ad-hoc Reporting System and supports flexible business reporting. The system also integrates with the Reference Data System, National Recreational Licensing System, National Special License Issue System and PacAdmin.

Custom Web Application Development

The Analysis module was developed using .NET, R, RODBC and R-serve to allow catch estimate generation. Using R language decoupled the statistical business logic from the application, allowing those with R experience to update the complex calculations and business logic without needing to modify the actual KREST source code. Using R-serve allowed the system to execute multiple analysis jobs within a fraction of the time and allowed the system to scale and meet the needs of stakeholders. Regional consistency, ability to create multiple versions of estimates, create workflow and to manage the promotion of results across estimates make the system far more flexible and productive than the previous system.

Benefits and Results

These changes resulted in several benefits:

  • Collected data adhered to and was stored in a standardized way, yet was flexible enough to accommodate the unique business requirements of different Fisheries within the region.
  • Web services provided a standardized approach data exchange between data collection systems and KREST.
  • Adding validation and enforcing data standards ensured collected data met minimum data requirements.
  • Developers of remote data entry tools had a standardized API to develop against. This ensured consistency in data quality and provided a single source for reference and meta data through one standard interface. Current systems leveraging this interface include: CREEL Remote System, Harvest Management System, MERP, eLog, Yukon Harvest and Conservation System, and National Recreational Licensing System.
  • The improvements in data collection, storage and processing created an increased confidence in the data and thus improved business confidence in the analysis and reports, recommendations and insights garnered.

One of the most significant challenges with this project has been the transformation and migration of over 20 years of legacy data to adhere to current data standards and ensure accurate mapping to the new KREST data schema. Bayleaf has worked with Fisheries since 2011 on this ambitious project which was completed in late 2015. At that time we shifted from a development role to a support role, as Fisheries transitioned from the old system to the new KREST system.