Apache Nifi release 0.6.0 and 0.6.1 recently and I wanted to go throught and highlight some of the key changes that may affect you Nifi users. So far Nifi has kept pretty well to thier 6 week release schedule that they have discussed. That’s a really agressive release schedule so we’ll see if they can keep that up once they hit the 0.7.0/1.0.0 release, which they say is going to actually be pushed back a little. The 0.6.0 release was again substantial with 52 bug fixes, 15 improvements and 7 new features. 0.6.1 didn’t take to long to follow with 11 bug fixes and 1 improvement. The release binaries are on the Apache Nifi download pageThe full release notes are avaialble on Apaches Nifi’s wiki page. The combined 0.6.* release information is below.
This release is substantial and really expands Apache Nifi’s usablity, specificall for those users who don’t feel comfortable with Java. Lets take a deeper look at what else is included!
Controller services are shared between processors, other controller services and reporting tasks. Normally they provide access to a shared resource, such as a database or ssl context, or externally managed content. This post will cover the basics of a controller service through a simple example. This example will take a file path that contains one or more properties files, and provide a processor access to those properties. The full source is hosted on Github.
Apache Nifi kicked out thier second release since graduating this past Friday. With the release of 0.4.0, they are one step closer to a 1.0.0 release which will contain some very interesting things. While we look forward to the first big release of Apache Nifi, lets break down the changes in the current build. As always, release 0.4.0 is available on Apache Nifi’s download page.
The release notes are available on Nifi’s confluence page and a quick highlight can be found below. First lets just say, this is the biggest release so far, with almost double the amount of bug fixes, and over double the amount of improvements and new features!
- General UI improvements to usability
- Multiple Authentication Mechanisms
- New Provenance event types and ability to searh provenance events
- Idle CPU usage was reduced
- New Processors!
- Improved OS support
Apache Nifi released their third release, 0.3.0 since graduating to a TLP. It was a minor release but still included a descent number of changes.
Release Highlights according to nifi can be found on their confluence page, but are also listed below.
- Performance improvements in handling large volumes of small files.
- Performance improvements in Provenance repositories.
- Added Reporting Task for ApacheTM AmbariTM.
- Improved stability of nifi bootstrap.
- Added Processors for working with images.
- Support for interacting with Kerberos enabled Hadoop clusters
- Added additional Avro capabilities - merging datafiles & converting to json
- Added processors for performing INSERT, UPDATE, DELETE statements against relational databases
Apache Nifi graduated from a podling to a top level Apache Software Foundation project a few months ago, which is great news! Congratulations to the NiFi team and all the contributors to making Apache Nifi great and meeting the requirements to move from an incubating project to a top level project.
The Apache Nifi framework has built in unit testing with Junit using test runners. They have a few examples in their code base, but learning first hand really helps. In this post we’ll go over adding unit tests to the JSON Processor that we developed previously.
To start, we’ll checkout the JSON Processor code from Github and then open it up in your favorite text editor. There is already a test package/folder in the project that contains a unit test, rocks.nifi.examples.processors/JsonProcessorTest.java.
When unit testing in Apache Nifi, there are a few items on top of the normal JUnit annotations that are required. While you can unit test with just JUnit, using the built in method in Apache Nifi makes it much easier. In a future post we’ll show you how to unit test using Mockito and JUnit to test helper methods where actually invoking a full processor seems excessive, or if you just don’t want to use Nifi’s test runner.
Apache Nifi just kicked out their second release, 0.0.2. It doesn’t have too many new features in it, hence a patch release according to semantic versioning, but it is definitely an improvement over the previous version, with it’s bug fixes, improvements, and new features numbers below.
The list of available Apache Nifi processors is extensive, as documented in this post. There is still a need to develop your own; to pull data from a database, to process an uncommon file format, or many other unique situations. So to get you started, we will work through a basic processor that takes a json file as input and a json path as a parameter to place into the contents and an attribute. The full source is hosted on Github.
Includes all processors through release
With new releases of Nifi, the number of processors have increased from the original 53 to 154 to what we currently have today! Here is a list of all processors, listed alphabetically, that are currently in Apache Nifi as of the most recent release. Each one links to a description of the processor further down. The Usage documentation available in the web ui has much more detail about each processor, it’s properties, modifiable attributes, and relationships and each processor has it’s own page in the UI, so here is just a quick overview. Again, this content is taken directly from Nifi’s Usage guide in their web UI and all credit/rights belong to them under the Apache 2.0 License.
Nifi has improved their documentation, which was originally only available when running apache nifi. The documentation now is produced through the build process and has been added to their website. So if you need more information or more detail about each processor just check there.
subscribe via RSS