Building Airflow Pipelines and Automated Data Extraction

We developed an Airflow Pipeline to extract text from unstructured documents and upload it to an elastic search database and created an external API for users outside of the team network. The Airflow pipeline automated workflows and managed boring and redundant manual tasks.

We built airflow pipelines for

• Extracting data from uploaded and emailed document

• Extracting companies investor presentations and transcript

• Extracting information from financial documents

• Extracting news data from peer websites.

We added a new feature where anyone from the organization can compare financial data on a quarterly basis or look into the yearly growth.

We did this by designing elastic search mapping and queries for refined results. Now anyone can ask questions like: What was NRG’s thirdquarter revenue growth percentage and they will get the reply at the same moment in a conversational way.

The more questions you ask the assistant - the smarter it gets. Another feature added by our team is to develop a process where the assistant nudges you with questions that you didn t think of asking. This can provide relevant insights and, hopefully, uncover new knowledge from the data.

The team also reinvented the backend data structure with dependency injection and storage abstraction. - increase in speed, reduced latency, restructured the backend using service based architecture.

We also redesigned Data Explorer for viewing different connected data sources in an improved manner. We improved table extraction logic for extracting tables from unstructured PDFs and documents on top of existing libraries ( Airflow )