This Community site is new! Please help us build a community around the JNIOR.
Sign up, and help share your knowledge. Please sign-up even if you do not plan to post as a sign of support.
If there is evidence of a demand we will continue to develop the content here.

The Art of Filing

Tips for programming on the JNIOR
Post Reply
Posts: 395
Joined: Thu Sep 14, 2017 12:55 pm

The Art of Filing

Post by bscloutier » Fri Jan 26, 2018 9:36 am

Digital Files are a given entity in the programming world. They can contain 0 to many bytes each of 256 values. Those could be ordered to represent everything from common text (ASCII) to binary bit streams (compressed data) and everything in between. Files therefore have a size and they also have a timestamp. These days that timestamp represents the date and time of the last modification to the file. They also carry permissions which can control who can access the file or even know of its existence.

There is also location. In part that means where the file is positioned within a directory or folder structure. We are also concerned with the type of media in which the file is stored. You know, did you put the file on a memory stick, on the hard drive or in the Cloud? This aspect of "location" is what we are going to consider here in this topic. Files stored on your JNIOR end up in one of four different areas but yet all appear to be in the same place. Where your file is located can affect performance and the longevity of the your data.

Each JNIOR contains multiple memory components each of which can provide for file storage. These are integrated into a single File System. There are 4 types of storage: RAM, DRAM, Flash and ROM. You actually utilize them all in routine operation. Let's look into it.

Non-volatile Battery-Backed Static Random Access Memory (SRAM)

When you enter the JNIOR's Console (Command Line Interface) either through the serial port, using Telnet or by opening the DCP, your working directory is the root of the File System or "/". By performing a DIR/LS command with the -L option you see content details generally containing the system's basic log files. You can also see that there are a number of sub-directories or sub-folders. I struggle with terminology here. Do you use "directory" or "folder"? I think that I haphazardly vacillate between the two.
bruce_dev /> dir -l
total 10
drwxrwxrwx   1 root      root           8 Jan 26 08:22 .
drwxrwxrwx   1 root      root           8 Jan 26 08:22 ..
dr-xr-xr-x   1 root      root           1 Dec 31 1999  etc
drwxr-xr-x   1 root      root          58 Jan 26 07:38 flash
drwxrwxrwx   1 root      root           0 Jan 25 15:13 temp
-rw-r--r--   1 root      root       40968 Jan 26 08:13 jniorsys.log
-rw-r--r--   1 root      root         956 Jan 26 08:13 jniorboot.log
-rw-r--r--   1 root      root        1005 Jan 26 08:00 jniorboot.log.bak
-rw-r--r--   1 root      root       40302 Jan 26 07:38 web.log
-rw-r--r--   1 jnior     root       22434 Jan 25 14:53 manifest.json
  1763.2 KB available

bruce_dev />
Now you might immediately notice that there is only 1763 KB available. That's not very much! Is that it?

No. But the File System root is located in a 2MB SRAM. This content is protected from loss by a battery. In fact, the battery is there for this purpose and to retain the current time and date during power outage. We built some JNIORs with a more expensive 4MB part but eventually realized that it wasn't necessary. The bulk of your file storage will be located elsewhere.

The advantage of the SRAM is it's speed and re-usability. In addition to the file system root, JANOS locates the Registry and other immutable memory blocks here. But space here is limited and it is best to preserve this area for system use. Data stored here does come with the risk of loss. This is a small probability but not an insignificant one. First of all the battery could die. If your JNIOR is powered 24/7 the battery should be there for you for 10 years and more. But if you power down the JNIOR routinely you may get 5 or 6 years out of it. Thankfully the Series 4 batteries are replaceable and you can get them at your local convenience store. Some customers though are happy to leave the dead battery not caring if their root folder is then volatile.

Perhaps more likely is that you decide to wipe the memory. You may have an application issue that gets the system into a problem condition. It is possible and we might recommend that you "business card' the battery. So by that we mean that you remove power from the unit, open it and slip a piece of something (business cards work well) under the battery tab for a few seconds. This clears the SRAM (and the clock). Typically you only lose the logs. The Registry and therefore your configuration is backed up by another file stored in another area. But don't worry, we recommend that procedure very very infrequently.

If you are programming your own JNIOR you might get yourself into a reboot loop. Basically your application starts up and performs something incorrectly that throws an assertion (system restart). The JNIOR reboots and restarts your application and another assertion ensues. Okay, not a great situation. JANOS eventually will detect some forms of reboot looping and it may decide to reformat the SRAM as a last ditch effort to restore access to your JNIOR. It sounds terrible but again it is a very very rare thing.

The point is that data stored in the root of the file system offers good performance and immediate data retention. It is not your best choice for long term storage. For that you want to use the Flash memory.

Posts: 395
Joined: Thu Sep 14, 2017 12:55 pm

Re: The Art of Filing

Post by bscloutier » Fri Jan 26, 2018 10:13 am

Flash File System

Flash memory retains data even in the absence of power. Files written in Flash memory are therefore retained even when the battery is removed. For that reason it is the best location for long term data storage. This is where you should place all of your programs, web site files and whatever else needs to be kept around. Everything under the /flash directory/folder is located in Flash memory.
bruce_dev /> dir -l flash
total 60
drwxr-xr-x   1 root      root          58 Jan 26 07:38 .
drwxrwxrwx   1 root      root           8 Jan 26 08:22 ..
drwxr-xr-x   1 root      root           1 Dec 06 11:15 cinema_backup
drwxr-xr-x   1 jnior     root           2 Dec 10 2015  generators
drwxr-xr-x   1 root      root           1 Jan 15 09:31 logs
drwxr-xr-x   1 root      root           2 Jan 15 09:32 public
drwxr-xr-x   1 root      root           2 Feb 06 2017  somepath
drwxr-xr-x   1 jnior     root          25 Jan 26 07:03 www
-rwxr-xr-x   1 jnior     root        1081 Jan 26 07:37 JTest.jar
-rw-r--r--   1 jnior     root       22434 Jan 25 14:53 manifest.json
-rw-r--r--   1 root      root        5449 Jan 23 08:33 jnior.ini
-rw-r--r--   1 jnior     root          13 Jan 11 15:16 gogo.dat
-rw-r--r--   1 jnior     root      183358 Jan 11 09:44
-rwxr-xr-x   1 jnior     root        3043 Jan 05 10:21 JTest2.jar
-rw-r--r--   1 jnior     root         278 Dec 12 13:28 pubkey.pem
-rw-r--r--   1 jnior     root        1092 Dec 08 12:48 honeypot.cer
-rw-r--r--   1 jnior     root         272 Dec 06 13:27
-rwxr-xr-x   1 jnior     root       20266 Dec 06 09:31 Cinekey.jar
-rwxr-xr-x   1 jnior     root      313835 Dec 04 13:44 Cinema.jar
-rwxr-xr-x   1 jnior     root        8329 Nov 21 12:04 Hmi.jar
-rwxr-xr-x   1 jnior     root        2189 Oct 04 14:24 JScan.jar
-rwxr-xr-x   1 jnior     root        3201 Sep 29 15:33 JUptime.jar
-rwxr-xr-x   1 jnior     root       58619 Aug 08 15:05 ModbusServer.jar
-rwxr-xr-x   1 jnior     root        4476 Jul 20 2017  Dmx.jar
-rw-r--r--   1 jnior     root         304 May 18 2017  test.txt
-rwxr-xr-x   1 jnior     root      169011 Apr 24 2017  snmp.jar
-rw-r--r--   1 jnior     root        1041 Feb 28 2017  key.pem
-rw-r--r--   1 jnior     root         902 Feb 15 2017  bruce_dev.cer
-rwxr-xr-x   1 root      root        4820 Jan 30 2017  jAccess.jar
-rwxr-xr-x   1 root      root        2174 Jan 23 2017  jPing.jar
-rwxr-xr-x   1 root      root        5651 Jan 23 2017  JManifest.jar
-rwxr-xr-x   1 root      root        1510 Dec 22 2016  ctrlc.jar
-rwxr-xr-x   1 jnior     root       74743 Oct 10 2016  Environ.jar
-rwxr-xr-x   1 jnior     root        9680 Oct 06 2016  ftp.jar
-rwxr-xr-x   1 jnior     root        4180 Aug 16 2016  TimeSearch.jar
-rwxr-xr-x   1 jnior     root        2616 Aug 03 2016  clktest.jar
-rwxr-xr-x   1 jnior     root       13079 Jul 27 2016  rz.jar
-rwxr-xr-x   1 jnior     root        2992 Jul 19 2016  Display.jar
-rwxr-xr-x   1 jnior     root       95325 Jun 30 2016  Buffer.jar
-rwxr-xr-x   1 jnior     root      112411 Jun 08 2016  slaveservice.jar
-rwxr-xr-x   1 jnior     root        5811 Jun 07 2016  UdpTest.jar
-rwxr-xr-x   1 jnior     root        5580 Jun 06 2016  jModule.jar
-rwxr-xr-x   1 jnior     root         969 Jun 02 2016  IntelliJ.jar
-rwxr-xr-x   1 jnior     root        1903 Jun 02 2016  Benchmark.jar
-rwxr-xr-x   1 jnior     root        4532 Mar 08 2016  SerialTest.jar
-rw-r--r--   1 root      root         898 Feb 10 2016  current.key
-rwxr-xr-x   1 jnior     root       32187 Dec 17 2015  serialcontrol.jar
-rwxr-xr-x   1 jnior     root      106794 Dec 10 2015  Utility.jar
-rwxr-xr-x   1 jnior     root      163902 Sep 04 2015  AnalogPresets.jar
-rwxr-xr-x   1 jnior     root        5053 Jul 28 2015  0-10vtest.jar
-rw-r--r--   1 jnior     root         898 Jul 24 2015  jnior1024.key
-rwxr-xr-x   1 jnior     root          56 Jul 10 2015  clean.bat
-rwxr-xr-x   1 jnior     root          17 Jun 30 2015  dirs.bat
-rwxr-xr-x   1 jnior     root        3862 Jun 18 2015  Test4to20.jar
-rwxr-xr-x   1 jnior     root       46590 Jun 18 2015  task.jar
-rwxr-xr-x   1 jnior     root        3601 Jun 18 2015  ThreadTest.jar
-rw-r--r--   1 jnior     root        4311 Jun 08 2015  task.ini
-rwxr-xr-x   1 jnior     root       25266 Jun 05 2015  serialethernet.jar
-rwxr-xr-x   1 jnior     root        2993 Jul 12 2013  4routtest.jar
-rwxr-xr-x   1 jnior     root        3142 Jan 17 2013  jPanel.jar
  26.85 MB flash available

bruce_dev />
Okay so my development unit is full of all kinds of stuff. Here you will notice that even so there is some 26 MB of file storage available. For the JNIOR that is a lot. You aren't dealing with large graphics files and such on the JNIOR. But if you were to develop a really sophisticated website hosted by the JNIOR you might fill that. If that is the case you might want the new 412DMX.
412dmx_r00 /> dir -l flash
total 22
drwxr-xr-x   1 root      root          20 Jan 11 09:45 .
drwxrwxrwx   1 root      root          16 Jan 23 13:27 ..
drwxr-xr-x   1 jnior     root          13 Oct 17 14:06 www
-rw-r--r--   1 jnior     root      183358 Jan 11 09:45
-rw-r--r--   1 root      root        2055 Dec 12 15:16 jnior.ini
-rwxr-xr-x   1 jnior     root        4526 Dec 05 14:15 Dmx.jar
-rwxr-xr-x   1 jnior     root        1597 Nov 17 07:34 ident.jar
-rw-r--r--   1 jnior     root       15584 Nov 07 09:04 manifest.json
-rw-r--r--   1 jnior     root       46000 Oct 12 12:37 string-test.dat
-rw-r--r--   1 jnior     root       20000 Oct 12 12:37 four-byte-test.dat
-rwxr-xr-x   1 jnior     root       42138 Oct 12 12:36 Benchmark.jar
-rw-r--r--   1 jnior     root       65481 Oct 11 11:10 lorem-ipsum.txt
-rwxr-xr-x   1 jnior     root       37110 Oct 05 14:51 MidNiteSolar.jar
-rwxr-xr-x   1 jnior     root       98569 Oct 03 13:51 ModbusClasses.jar
-rwxr-xr-x   1 jnior     root       58620 Oct 03 13:45 ModbusServer.jar
-rwxr-xr-x   1 jnior     root        3971 Oct 03 13:45 Simulator.jar
-rwxr-xr-x   1 jnior     root       95488 May 08 2017  SNMP.jar
-rwxr-xr-x   1 jnior     root      115448 May 08 2017  task.jar
-rwxr-xr-x   1 jnior     root       54247 Feb 03 2017  SlaveService.jar
-rwxr-xr-x   1 jnior     root       87637 Feb 03 2017  serialethernet.jar
-rwxr-xr-x   1 jnior     root       31640 Feb 03 2017  serialcontrol.jar
-rwxr-xr-x   1 jnior     root        9563 Feb 03 2017  ftp.jar
  509.70 MB flash available

412dmx_r00 /> 
Here there is close to 1/2 GB of file space. Actually we will be shipping the 412DMX with 1/4 GB capacity.

The existing JNIOR line uses a 32 MB serial Flash component. Data is written to and read from this Flash device using a serial (SPI) channel. This memory is therefore slower. This is not an issue though as JANOS uses a sophisticated caching system to handle Flash I/O. And if power is lost in the midst of a lengthy Flash write the device's integrity is not damaged. The JANOS Flash File System uses a fault tolerant form of transaction processing. In the event of power loss (or crash) the Flash File System rolls back to the last known good configuration. As a result data stored here is likely to remain until purposely deleted. You can reformat the Flash File System but generally there is hardly ever a need to do so.

The 412DMX introduces a different Flash technology to the line. Here we employ a parallel NAND Flash memory. In addition to greater capacity the read and write access timing has significantly improved. Files stored here are accessed with almost the same performance as SRAM. In fact, in the future we may move the File System root to Flash and eliminate the SRAM altogether. Potentially the NAND Flash can be implemented on the 410, 421 and 414 and it will be considered when PCB revisions occur on those models.

Posts: 395
Joined: Thu Sep 14, 2017 12:55 pm

Re: The Art of Filing

Post by bscloutier » Fri Jan 26, 2018 10:51 am

Temporary Storage

Files stored in the /temp folder are considered temporary. That folder is actually located in the Heap which as I mentioned is DRAM memory. That memory is reformatted on boot. So the /temp folder always comes up being empty.
bruce_dev /> dir -l /temp
total 2
drwxrwxrwx   1 root      root           0 Jan 25 15:13 .
drwxrwxrwx   1 root      root           8 Jan 26 08:22 ..
  62.87 MB available (temporary)

bruce_dev />
The JNIORs are shipping now with 64 MB of Heap memory. The system normally utilizes only about 3 or 4 MB of that. So the /temp folder has reasonable capacity. This is twice what is available in the standard JNIOR Flash but much less than will be available in the 412DMX Flash. This is a great place to create temporary files. This provides the best performance as well.

We recommend that you transfer UPD files for updates first to the /temp folder. The advantage being that the file disappears once the update has been completed. UPD files are quite large and generally don't fit into the File System root. You certainly wouldn't want to leave one in the root for very long. And placing the UPD in Flash is not necessary and slow to accomplish.

An application might first create a file here and should the procedure complete properly then move it to long term storage. This is also great for files that will be accessed randomly (using a lot of fseek). You might improve an application's performance by copying a database to /temp first. It would remain until reboot. Of course that is heap memory and the same memory where a large byte buffer would be allocated. So to improve performance an application might read the entire file into a byte buffer and access that directly. The load on the heap would be the same and random access would be greatly simplified.

The /etc Folder

Lastly there is the /etc folder. This is not a writable area and it is actually built into JANOS. This is where JANOS provides system files as might be necessary for application execution. That is the case now for the JanosClasses.jar file.
bruce_dev /> dir -l /etc
total 3
dr-xr-xr-x   1 root      root           1 Dec 31 1999  .
drwxrwxrwx   1 root      root           8 Jan 26 08:22 ..
-r-----r--   1 root      root      266601 Jan 11 09:58 JanosClasses.jar
  0 KB available (read only)

bruce_dev />
So since this is read-only there is no space available. This is stored within the processor in its Program ROM. Access is very fast.

It is important though as you can download this JAR file and use it in compiling your applications for the JNIOR. I would recommend getting the JAR from us or this site that not only contains these classes but source stubs and JavaDoc as well. Clearly that would help you more in development.

JNIORs are shipped with a number of default files in /flash. Some of those should be updated when JANOS is updated. In the future there may be additional files included in /etc. So it is something to keep an eye on.

In Summary

The JANOS File System appears to be centrally located and of a single directory structure. Yet it covers storage in a variety of media. One needs to keep this in mind when deciding where to place files either for temporary use or long term availability. Files in different areas experience different performance levels and different risks. Keeping this in mind you can better manage your JNIOR controller and create great applications.

Post Reply