Open Source Etl Framework

Open Source ETL framework

I've used Kettle. It has its own GUI, but if you rather use the API to do the ETL yourself it's also supported. It has proved to be very useful to me and there are a few plugins already available for it.

Java Opensource ETL framework with automatic scheduling

Use Jenkins. Most people use it for deployment and software builds, but honestly it is really a super cron. It is perfect for periodic jobs. For ETL use what you think is best: Talend, Clover, Pentaho, scripting language, etc. Just make sure whatever you package up can be run via the command line so you can use Jenkins.

ETL Tools and Build Tools

It is correct that you can roll-out your own ETL scripts written using a development tool of your preference. Having said that, ETL jobs are frequently large (for a lack of better word) and demand considerable administration and attention to minute details (like programming). ETL tools allow developer to focus on ETL tasks -- as opposed to writing and debugging code, although that's part of it too. There are some open-source tools out there, so you can get a feeling of what an average tool does, before jumping into custom development. For example, more expensive tools provide data lineage, meaning you can (graphically) track every field on a report back to the originating table through all transformations (versions included); after a corporate merger that's quite a task to do.

For example Pentaho has community edition; if you have MS SQL Server, you can get SSIS. Also see if you can find something here.

Framework/ Library to compare/assert data across platforms

I would propose such tools as QuerySurge (https://www.querysurge.com/),

or ETL Validator (https://www.datagaps.com/etl-testing-tools/etl-validator/),

or any other tool that is mentioned in this link(https://www.guru99.com/etl-testing-tools.html),

however if you desire an open source data validation tool,
you should try https://github.com/GoogleCloudPlatform/professional-services-data-validator

Hope I've been helpful,

R

---

Related Topics