The Art of Filing January 26, 2018
Digital Files are a given entity in the programming world. They can contain 0 to many bytes each of 256 values. Those could be ordered to represent everything from common text (ASCII) to binary bit streams (compressed data) and everything in between. Files therefore have a size and they also have a timestamp. These days that timestamp represents the date and time of the last modification to the file. They also carry permissions which can control who can access the file or even know of its existence.
There is also location. In part that means where the file is positioned within a directory or folder structure. We are also concerned with the type of media in which the file is stored. You know, did you put the file on a memory stick, on the hard drive or in the Cloud? This aspect of “location” is what we are going to consider here in this topic. Files stored on your JNIOR end up in one of four different areas but yet all appear to be in the same place. Where your file is located can affect performance and the longevity of the your data.
Each JNIOR contains multiple memory components each of which can provide for file storage. These are integrated into a single File System. There are 4 types of storage: RAM, DRAM, Flash and ROM. You actually utilize them all in routine operation. Let’s look into it.
Non-volatile Battery-Backed Static Random Access Memory (SRAM)
When you enter the JNIOR’s Console (Command Line Interface) either through the serial port, using Telnet or by opening the DCP, your working directory is the root of the File System or “/”. By performing a DIR/LS command with the -L option you see content details generally containing the system’s basic log files. You can also see that there are a number of sub-directories or sub-folders. I struggle with terminology here. Do you use “directory” or “folder”? I think that I haphazardly vacillate between the two.
bruce_dev /> dir -l total 10 drwxrwxrwx 1 root root 8 Jan 26 08:22 . drwxrwxrwx 1 root root 8 Jan 26 08:22 .. dr-xr-xr-x 1 root root 1 Dec 31 1999 etc drwxr-xr-x 1 root root 58 Jan 26 07:38 flash drwxrwxrwx 1 root root 0 Jan 25 15:13 temp -rw-r--r-- 1 root root 40968 Jan 26 08:13 jniorsys.log -rw-r--r-- 1 root root 956 Jan 26 08:13 jniorboot.log -rw-r--r-- 1 root root 1005 Jan 26 08:00 jniorboot.log.bak -rw-r--r-- 1 root root 40302 Jan 26 07:38 web.log -rw-r--r-- 1 jnior root 22434 Jan 25 14:53 manifest.json 1763.2 KB available bruce_dev />
Now you might immediately notice that there is only 1763 KB available. That’s not very much! Is that it?
No. But the File System root is located in a 2MB SRAM. This content is protected from loss by a battery. In fact, the battery is there for this purpose and to retain the current time and date during power outage. We built some JNIORs with a more expensive 4MB part but eventually realized that it wasn’t necessary. The bulk of your file storage will be located elsewhere.
The advantage of the SRAM is it’s speed and re-usability. In addition to the file system root, JANOS locates the Registry and other immutable memory blocks here. But space here is limited and it is best to preserve this area for system use. Data stored here does come with the risk of loss. This is a small probability but not an insignificant one. First of all the battery could die. If your JNIOR is powered 24/7 the battery should be there for you for 10 years and more. But if you power down the JNIOR routinely you may get 5 or 6 years out of it. Thankfully the Series 4 batteries are replaceable and you can get them at your local convenience store. Some customers though are happy to leave the dead battery not caring if their root folder is then volatile.
Perhaps more likely is that you decide to wipe the memory. You may have an application issue that gets the system into a problem condition. It is possible and we might recommend that you “business card’ the battery. So by that we mean that you remove power from the unit, open it and slip a piece of something (business cards work well) under the battery tab for a few seconds. This clears the SRAM (and the clock). Typically you only lose the logs. The Registry and therefore your configuration is backed up by another file stored in another area. But don’t worry, we recommend that procedure very very infrequently.
If you are programming your own JNIOR you might get yourself into a reboot loop. Basically your application starts up and performs something incorrectly that throws an assertion (system restart). The JNIOR reboots and restarts your application and another assertion ensues. Okay, not a great situation. JANOS eventually will detect some forms of reboot looping and it may decide to reformat the SRAM as a last ditch effort to restore access to your JNIOR. It sounds terrible but again it is a very very rare thing.
The point is that data stored in the root of the file system offers good performance and immediate data retention. It is not your best choice for long term storage. For that you want to use the Flash memory.
Flash File System
Flash memory retains data even in the absence of power. Files written in Flash memory are therefore retained even when the battery is removed. For that reason it is the best location for long term data storage. This is where you should place all of your programs, web site files and whatever else needs to be kept around. Everything under the /flash directory/folder is located in Flash memory.
bruce_dev /> dir -l flash total 60 drwxr-xr-x 1 root root 58 Jan 26 07:38 . drwxrwxrwx 1 root root 8 Jan 26 08:22 .. drwxr-xr-x 1 root root 1 Dec 06 11:15 cinema_backup drwxr-xr-x 1 jnior root 2 Dec 10 2015 generators drwxr-xr-x 1 root root 1 Jan 15 09:31 logs drwxr-xr-x 1 root root 2 Jan 15 09:32 public drwxr-xr-x 1 root root 2 Feb 06 2017 somepath drwxr-xr-x 1 jnior root 25 Jan 26 07:03 www -rwxr-xr-x 1 jnior root 1081 Jan 26 07:37 JTest.jar -rw-r--r-- 1 jnior root 22434 Jan 25 14:53 manifest.json -rw-r--r-- 1 root root 5449 Jan 23 08:33 jnior.ini -rw-r--r-- 1 jnior root 13 Jan 11 15:16 gogo.dat -rw-r--r-- 1 jnior root 183358 Jan 11 09:44 www.zip -rwxr-xr-x 1 jnior root 3043 Jan 05 10:21 JTest2.jar -rw-r--r-- 1 jnior root 278 Dec 12 13:28 pubkey.pem -rw-r--r-- 1 jnior root 1092 Dec 08 12:48 honeypot.cer -rw-r--r-- 1 jnior root 272 Dec 06 13:27 key.pub -rwxr-xr-x 1 jnior root 20266 Dec 06 09:31 Cinekey.jar -rwxr-xr-x 1 jnior root 313835 Dec 04 13:44 Cinema.jar -rwxr-xr-x 1 jnior root 8329 Nov 21 12:04 Hmi.jar -rwxr-xr-x 1 jnior root 2189 Oct 04 14:24 JScan.jar -rwxr-xr-x 1 jnior root 3201 Sep 29 15:33 JUptime.jar -rwxr-xr-x 1 jnior root 58619 Aug 08 15:05 ModbusServer.jar -rwxr-xr-x 1 jnior root 4476 Jul 20 2017 Dmx.jar -rw-r--r-- 1 jnior root 304 May 18 2017 test.txt -rwxr-xr-x 1 jnior root 169011 Apr 24 2017 snmp.jar -rw-r--r-- 1 jnior root 1041 Feb 28 2017 key.pem -rw-r--r-- 1 jnior root 902 Feb 15 2017 bruce_dev.cer -rwxr-xr-x 1 root root 4820 Jan 30 2017 jAccess.jar -rwxr-xr-x 1 root root 2174 Jan 23 2017 jPing.jar -rwxr-xr-x 1 root root 5651 Jan 23 2017 JManifest.jar -rwxr-xr-x 1 root root 1510 Dec 22 2016 ctrlc.jar -rwxr-xr-x 1 jnior root 74743 Oct 10 2016 Environ.jar -rwxr-xr-x 1 jnior root 9680 Oct 06 2016 ftp.jar -rwxr-xr-x 1 jnior root 4180 Aug 16 2016 TimeSearch.jar -rwxr-xr-x 1 jnior root 2616 Aug 03 2016 clktest.jar -rwxr-xr-x 1 jnior root 13079 Jul 27 2016 rz.jar -rwxr-xr-x 1 jnior root 2992 Jul 19 2016 Display.jar -rwxr-xr-x 1 jnior root 95325 Jun 30 2016 Buffer.jar -rwxr-xr-x 1 jnior root 112411 Jun 08 2016 slaveservice.jar -rwxr-xr-x 1 jnior root 5811 Jun 07 2016 UdpTest.jar -rwxr-xr-x 1 jnior root 5580 Jun 06 2016 jModule.jar -rwxr-xr-x 1 jnior root 969 Jun 02 2016 IntelliJ.jar -rwxr-xr-x 1 jnior root 1903 Jun 02 2016 Benchmark.jar -rwxr-xr-x 1 jnior root 4532 Mar 08 2016 SerialTest.jar -rw-r--r-- 1 root root 898 Feb 10 2016 current.key -rwxr-xr-x 1 jnior root 32187 Dec 17 2015 serialcontrol.jar -rwxr-xr-x 1 jnior root 106794 Dec 10 2015 Utility.jar -rwxr-xr-x 1 jnior root 163902 Sep 04 2015 AnalogPresets.jar -rwxr-xr-x 1 jnior root 5053 Jul 28 2015 0-10vtest.jar -rw-r--r-- 1 jnior root 898 Jul 24 2015 jnior1024.key -rwxr-xr-x 1 jnior root 56 Jul 10 2015 clean.bat -rwxr-xr-x 1 jnior root 17 Jun 30 2015 dirs.bat -rwxr-xr-x 1 jnior root 3862 Jun 18 2015 Test4to20.jar -rwxr-xr-x 1 jnior root 46590 Jun 18 2015 task.jar -rwxr-xr-x 1 jnior root 3601 Jun 18 2015 ThreadTest.jar -rw-r--r-- 1 jnior root 4311 Jun 08 2015 task.ini -rwxr-xr-x 1 jnior root 25266 Jun 05 2015 serialethernet.jar -rwxr-xr-x 1 jnior root 2993 Jul 12 2013 4routtest.jar -rwxr-xr-x 1 jnior root 3142 Jan 17 2013 jPanel.jar 26.85 MB flash available bruce_dev />
Okay so my development unit is full of all kinds of stuff. Here you will notice that even so there is some 26 MB of file storage available. For the JNIOR that is a lot. You aren’t dealing with large graphics files and such on the JNIOR. But if you were to develop a really sophisticated website hosted by the JNIOR you might fill that. If that is the case you might want the new 412DMX.
412dmx_r00 /> dir -l flash total 22 drwxr-xr-x 1 root root 20 Jan 11 09:45 . drwxrwxrwx 1 root root 16 Jan 23 13:27 .. drwxr-xr-x 1 jnior root 13 Oct 17 14:06 www -rw-r--r-- 1 jnior root 183358 Jan 11 09:45 www.zip -rw-r--r-- 1 root root 2055 Dec 12 15:16 jnior.ini -rwxr-xr-x 1 jnior root 4526 Dec 05 14:15 Dmx.jar -rwxr-xr-x 1 jnior root 1597 Nov 17 07:34 ident.jar -rw-r--r-- 1 jnior root 15584 Nov 07 09:04 manifest.json -rw-r--r-- 1 jnior root 46000 Oct 12 12:37 string-test.dat -rw-r--r-- 1 jnior root 20000 Oct 12 12:37 four-byte-test.dat -rwxr-xr-x 1 jnior root 42138 Oct 12 12:36 Benchmark.jar -rw-r--r-- 1 jnior root 65481 Oct 11 11:10 lorem-ipsum.txt -rwxr-xr-x 1 jnior root 37110 Oct 05 14:51 MidNiteSolar.jar -rwxr-xr-x 1 jnior root 98569 Oct 03 13:51 ModbusClasses.jar -rwxr-xr-x 1 jnior root 58620 Oct 03 13:45 ModbusServer.jar -rwxr-xr-x 1 jnior root 3971 Oct 03 13:45 Simulator.jar -rwxr-xr-x 1 jnior root 95488 May 08 2017 SNMP.jar -rwxr-xr-x 1 jnior root 115448 May 08 2017 task.jar -rwxr-xr-x 1 jnior root 54247 Feb 03 2017 SlaveService.jar -rwxr-xr-x 1 jnior root 87637 Feb 03 2017 serialethernet.jar -rwxr-xr-x 1 jnior root 31640 Feb 03 2017 serialcontrol.jar -rwxr-xr-x 1 jnior root 9563 Feb 03 2017 ftp.jar 509.70 MB flash available 412dmx_r00 />
Here there is close to 1/2 GB of file space. Actually we will be shipping the 412DMX with 1/4 GB capacity.
The existing JNIOR line uses a 32 MB serial Flash component. Data is written to and read from this Flash device using a serial (SPI) channel. This memory is therefore slower. This is not an issue though as JANOS uses a sophisticated caching system to handle Flash I/O. And if power is lost in the midst of a lengthy Flash write the device’s integrity is not damaged. The JANOS Flash File System uses a fault tolerant form of transaction processing. In the event of power loss (or crash) the Flash File System rolls back to the last known good configuration. As a result data stored here is likely to remain until purposely deleted. You can reformat the Flash File System but generally there is hardly ever a need to do so.
The 412DMX introduces a different Flash technology to the line. Here we employ a parallel NAND Flash memory. In addition to greater capacity the read and write access timing has significantly improved. Files stored here are accessed with almost the same performance as SRAM. In fact, in the future we may move the File System root to Flash and eliminate the SRAM altogether. Potentially the NAND Flash can be implemented on the 410, 421 and 414 and it will be considered when PCB revisions occur on those models.
Temporary Storage
Files stored in the /temp folder are considered temporary. That folder is actually located in the Heap which as I mentioned is DRAM memory. That memory is reformatted on boot. So the /temp folder always comes up being empty.
bruce_dev /> dir -l /temp total 2 drwxrwxrwx 1 root root 0 Jan 25 15:13 . drwxrwxrwx 1 root root 8 Jan 26 08:22 .. 62.87 MB available (temporary) bruce_dev />
The JNIORs are shipping now with 64 MB of Heap memory. The system normally utilizes only about 3 or 4 MB of that. So the /temp folder has reasonable capacity. This is twice what is available in the standard JNIOR Flash but much less than will be available in the 412DMX Flash. This is a great place to create temporary files. This provides the best performance as well.
We recommend that you transfer UPD files for updates first to the /temp folder. The advantage being that the file disappears once the update has been completed. UPD files are quite large and generally don’t fit into the File System root. You certainly wouldn’t want to leave one in the root for very long. And placing the UPD in Flash is not necessary and slow to accomplish.
An application might first create a file here and should the procedure complete properly then move it to long term storage. This is also great for files that will be accessed randomly (using a lot of fseek). You might improve an application’s performance by copying a database to /tempfirst. It would remain until reboot. Of course that is heap memory and the same memory where a large byte buffer would be allocated. So to improve performance an application might read the entire file into a byte buffer and access that directly. The load on the heap would be the same and random access would be greatly simplified.
The /etc Folder
Lastly there is the /etc folder. This is not a writable area and it is actually built into JANOS. This is where JANOS provides system files as might be necessary for application execution. That is the case now for the JanosClasses.jar file.
bruce_dev /> dir -l /etc total 3 dr-xr-xr-x 1 root root 1 Dec 31 1999 . drwxrwxrwx 1 root root 8 Jan 26 08:22 .. -r-----r-- 1 root root 266601 Jan 11 09:58 JanosClasses.jar 0 KB available (read only) bruce_dev />
So since this is read-only there is no space available. This is stored within the processor in its Program ROM. Access is very fast.
It is important though as you can download this JAR file and use it in compiling your applications for the JNIOR. I would recommend getting the JAR from us or this site that not only contains these classes but source stubs and JavaDoc as well. Clearly that would help you more in development.
JNIORs are shipped with a number of default files in /flash. Some of those should be updated when JANOS is updated. In the future there may be additional files included in /etc. So it is something to keep an eye on.
In Summary
The JANOS File System appears to be centrally located and of a single directory structure. Yet it covers storage in a variety of media. One needs to keep this in mind when deciding where to place files either for temporary use or long term availability. Files in different areas experience different performance levels and different risks. Keeping this in mind you can better manage your JNIOR controller and create great applications.
By Bruce Cloutier
| On January 26, 2018 10:26 am
| No Comments
| Categorized in: Embedded Programming 
@integpg