Handles converting buffers to and from disk. More...
#include <CubeIoHandler.h>


Classes | |
| class | BufferToChunkWriter |
| This class is designed to handle write() asynchronously. More... | |
Public Member Functions | |
| CubeIoHandler (QFile *dataFile, const QList< int > *virtualBandList, const Pvl &label, bool alreadyOnDisk) | |
| Creates a new CubeIoHandler using a RegionalCachingAlgorithm. | |
| virtual | ~CubeIoHandler () |
| Cleans up all allocated memory. | |
| virtual void | read (Buffer &bufferToFill) const |
| Read cube data from disk into the buffer. | |
| virtual void | write (const Buffer &bufferToWrite) |
| Write buffer data into the cube data on disk. | |
| virtual void | addCachingAlgorithm (CubeCachingAlgorithm *algorithm) |
| This will add the given caching algorithm to the list of attempted caching algorithms. | |
| virtual void | clearCache (bool blockForWriteCache=true) const |
| Free all cube chunks (cached cube data) from memory and write them to disk. | |
| virtual BigInt | getDataSize () const |
| virtual void | updateLabels (Pvl &labels)=0 |
| Function to update the labels with a Pvl object. | |
| int | bandCount () const |
| int | getBandCountInChunk () const |
| BigInt | getBytesPerChunk () const |
| int | getChunkCountInBandDimension () const |
| int | getChunkCountInLineDimension () const |
| int | getChunkCountInSampleDimension () const |
| int | getChunkIndex (const RawCubeChunk &) const |
| Given a chunk, what's its index in the file. | |
| BigInt | getDataStartByte () const |
| int | lineCount () const |
| int | getLineCountInChunk () const |
| PixelType | pixelType () const |
| int | sampleCount () const |
| int | getSampleCountInChunk () const |
| void | setVirtualBands (const QList< int > *virtualBandList) |
| This changes the virtual band list. | |
| QMutex * | dataFileMutex () |
| Get the mutex that this IO handler is using around I/Os on the given data file. | |
Protected Member Functions | |
| QFile * | getDataFile () |
| void | setChunkSizes (int numSamples, int numLines, int numBands) |
| This should be called once from the child constructor. | |
| virtual void | readRaw (RawCubeChunk &chunkToFill)=0 |
| This needs to populate the chunkToFill with unswapped raw bytes from the disk. | |
| virtual void | writeRaw (const RawCubeChunk &chunkToWrite)=0 |
| This needs to write the chunkToWrite directly to disk with no modifications to the data itself. | |
Protected Attributes | |
| QMutex * | m_writeThreadMutex |
| This enables us to block while the write thread is working. | |
| QList< int > * | m_virtualBands |
| Converts from virtual band to physical band. | |
Private Member Functions | |
| CubeIoHandler (const CubeIoHandler &other) | |
| Disallow copying of this object. | |
| CubeIoHandler & | operator= (const CubeIoHandler &other) |
| Disallow assignments of this object. | |
| void | blockUntilThreadPoolEmpty () const |
| This blocks (doesn't return) until the number of active runnables in the thread pool goes to 0. | |
| QPair< QList< RawCubeChunk * >, QList< int > > | findCubeChunks (int startSample, int numSamples, int startLine, int numLines, int startBand, int numBands) const |
| Get the cube chunks that correspond to the given cube area. | |
| void | findIntersection (const RawCubeChunk &cube1, const Buffer &cube2, int &startX, int &startY, int &startZ, int &endX, int &endY, int &endZ) const |
| Find the intersection between the buffer area and the cube chunk. | |
| void | flushWriteCache (bool force=false) const |
| This attempts to write the so-far-unwritten buffers from m_writeCache into the cube's RawCubeChunk cache. | |
| void | freeChunk (RawCubeChunk *chunkToFree) const |
| If the chunk is dirty, then we write it to disk. | |
| RawCubeChunk * | getChunk (int chunkIndex, bool allocateIfNecessary) const |
| Retrieve the cached chunk at the given chunk index, if there is one. | |
| int | getChunkCount () const |
| void | getChunkPlacement (int chunkIndex, int &startSample, int &startLine, int &startBand, int &endSample, int &endLine, int &endBand) const |
| Get the X/Y/Z (Sample,Line,Band) range of the chunk at the given index. | |
| RawCubeChunk * | getNullChunk (int chunkIndex) const |
| This creates a chunk filled with NULLs whose placement is at chunkIndex's position. | |
| void | minimizeCache (const QList< RawCubeChunk * > &justUsed, const Buffer &justRequested) const |
| Apply the caching algorithms and get rid of excess cube data in memory. | |
| void | synchronousWrite (const Buffer &bufferToWrite) |
| This method takes the given buffer and synchronously puts it into the Cube's cache. | |
| void | writeIntoDouble (const RawCubeChunk &chunk, Buffer &output, int startIndex) const |
| Write the intersecting area of the chunk into the buffer. | |
| void | writeIntoRaw (const Buffer &buffer, RawCubeChunk &output, int index) const |
| Write the intersecting area of the buffer into the chunk. | |
| void | writeNullDataToDisk () const |
| Write all NULL cube chunks that have not yet been accessed to disk. | |
Static Private Member Functions | |
| static bool | bufferLessThan (Buffer *const &lhs, Buffer *const &rhs) |
| This is used for sorting buffers into the most efficient write order. | |
Private Attributes | |
| QFile * | m_dataFile |
| The file containing cube data. | |
| BigInt | m_startByte |
| The start byte of the cube data. | |
| PixelType | m_pixelType |
| The format of each DN in the cube. | |
| double | m_base |
| The additive offset of the data on disk. | |
| double | m_multiplier |
| The multiplicative factor of the data on disk. | |
| ByteOrder | m_byteOrder |
| The byte order (endianness) of the data on disk. | |
| EndianSwapper * | m_byteSwapper |
| A helper that swaps byte order to and from file order. | |
| int | m_numSamples |
| The number of samples in the cube. | |
| int | m_numLines |
| The number of lines in the cube. | |
| int | m_numBands |
| The number of physical bands in the cube. | |
| QList< CubeCachingAlgorithm * > * | m_cachingAlgorithms |
| The caching algorithms to use, in order of priority. | |
| QMap< int, RawCubeChunk * > * | m_rawData |
| The map from chunk index to chunk for cached data. | |
| QMap< int, bool > * | m_dataIsOnDiskMap |
| The map from chunk index to on-disk status, all true if not allocated. | |
| int | m_samplesInChunk |
| The number of samples in a cube chunk. | |
| int | m_linesInChunk |
| The number of lines in a cube chunk. | |
| int | m_bandsInChunk |
| The number of physical bands in a cube chunk. | |
| QList< RawCubeChunk * > * | m_lastProcessByLineChunks |
| This is an optimization for process by line. | |
| QByteArray * | m_nullChunkData |
| A raw cube chunk's data when it was all NULL's. Used for speed. | |
| QPair< QMutex *, QList< Buffer * > > * | m_writeCache |
| These are the buffers we need to write to raw cube chunks. | |
| QThreadPool * | m_ioThreadPool |
| This contains threads for doing cube writes (and later maybe cube reads). | |
| bool | m_lastOperationWasWrite |
| If the last operation was a write then we need to flush the cache when reading. | |
| bool | m_useOptimizedCubeWrite |
| This is true if the Isis preference for the cube write thread is optimized. | |
| volatile int | m_idealFlushSize |
| Ideal write cache flush size. | |
| int | m_consecutiveOverflowCount |
| How many times the write cache has overflown in a row. | |
Handles converting buffers to and from disk.
This class handles converting buffers to and from disk. This class holds the cube chunks in memory and is capable of reading and writing them. It asks the caching algorithms to recommend cube chunks to not keep in memory. Children need to call setChunkSizes() in their constructor.
This class handles all of the virtual band conversions. This class also guarantees that unwritten cube data ends up read and written as NULLs. The default caching algorithm is a RegionalCachingAlgorithm.
2011-06-27 Steven Lambright - Added addCachingAlgorithm()
2011-07-18 Jai Rideout and Steven Lambright - Added unimplemented copy constructor and assignment operator.
2011-07-18 Jai Rideout and Steven Lambright - getDataFile() is no longer const, along with readRaw() and writeRaw().
2011-11-23 Jai Rideout - Added fix to findCubeChunks() and findIntersection() so that requested areas that are outside of the virtual bands of the cube will no longer fail, but instead will fill the buffer with nulls.
2012-02-17 Steven Lambright - The read() method is now const - the caching should be transparent to the API. It is still important to note that you cannot call read() and write() simultaneously - Cube is guaranteeing this. Backgrounded writes are now implemented and fully supported. Timing tests show results like this: Before: User: 358.71 Kernel: 38.29 Total Elapsed: 6:49.83 After: User: 380.01 Kernel: 44.97 Total Elapsed: 6:19.96 User- user-space CPU time consumed. Kernel- kernel-space CPU time consumed. The CPU consumption and kernel time consumption goes up, but the overall run time is lower. This makes the option perfectly viable for typical multi-core desktop solutions. Turning off the cube write optimization results in equivalent run times as before implementation. These time tests were run before the adaptive cache flush size was added, which improved performance further. References #727.
2012-06-06 Jeff Anderson - The read() method was modified to improve the speed for small buffer sizes as seen in ProcessByBoxcar and ProcessRubbersheet. The internal cache was always checked to be minimized which is slow for every read. Now the cache is only mimimized if it changed in size. Reference #894.
2014-04-07 Kimberly Oyama and Stuart Sides - Modified the findCubeChunks, writeIntoDouble, and writeIntoRaw methods to handle repeating virtual bands. Fixes #1927.
2014-09-30 Ian Humphrey - Modified read method to correctly access virtual bands when cube dimensions and buffer shape are same size. Fixes #1689.
2015-01-30 Ian Humphrey - Modified destructor to free m_writThreadMutex to prevent memory leaks upon destruction. Fixes #2082.
2016-04-21 Makayla Shepherd - Added UnsignedWord pixel type handling.
2016-06-21 Kris Becker - Properly forward declare QPair as struct not class
2016-08-28 Kelvin Rodriguez - updateLabels now a pure virtual, it had no implementation causing warnings in clang. Part of OS X 10.11 porting. QPair forward declaration now properly claims it as a struct.
2017-09-22 Cole Neubauer - Fixed documentation. References #4807
2018-07-20 Tyler Wilson - Added support for unsigned integer special pixel values. in functions writeIntoRaw(...) and writeIntoDouble(...) References #971.
2018-08-13 Summer Stapleton - Fixed incoming buffer comparison values for unsigned int type in writeIntoRaw(...).
Definition at line 109 of file CubeIoHandler.h.
| Isis::CubeIoHandler::CubeIoHandler | ( | QFile * | dataFile, |
| const QList< int > * | virtualBandList, | ||
| const Pvl & | label, | ||
| bool | alreadyOnDisk ) |
Creates a new CubeIoHandler using a RegionalCachingAlgorithm.
The chunk sizes must be set by a child in its constructor.
| dataFile | The file that contains cube data. This should be a valid, opened QFile and may not be NULL. The file should have at least read permissions. |
| virtualBandList | A list where the indices are the vbands and the values are the physical bands. The values are 1-based. This can be specified as NULL, in which case the vbands are the physical bands. The virtual band list is copied (the pointer provided isn't remembered). |
| label | The label which contains the "Pixels" and "Core" groups. |
| alreadyOnDisk | True if the cube exists; false ensures all NULLs are initialized into the file before this object is destructed. |
Definition at line 61 of file CubeIoHandler.cpp.
References m_bandsInChunk, m_base, m_byteSwapper, m_cachingAlgorithms, m_consecutiveOverflowCount, m_dataFile, m_dataIsOnDiskMap, m_idealFlushSize, m_ioThreadPool, m_lastOperationWasWrite, m_lastProcessByLineChunks, m_linesInChunk, m_multiplier, m_nullChunkData, m_numBands, m_numLines, m_numSamples, m_pixelType, m_rawData, m_samplesInChunk, m_startByte, m_useOptimizedCubeWrite, m_writeCache, and Isis::ImageIoHandler::m_writeThreadMutex.
Referenced by Isis::CubeIoHandler::BufferToChunkWriter::BufferToChunkWriter(), clearCache(), Isis::CubeBsqHandler::CubeBsqHandler(), CubeIoHandler(), Isis::CubeTileHandler::CubeTileHandler(), flushWriteCache(), freeChunk(), getChunk(), operator=(), and writeNullDataToDisk().
|
virtual |
Cleans up all allocated memory.
Child destructors must call clearCache() because we can no longer do IO by the time this destructor is called.
Definition at line 152 of file CubeIoHandler.cpp.
References m_byteSwapper, m_cachingAlgorithms, m_dataIsOnDiskMap, m_ioThreadPool, m_lastProcessByLineChunks, m_nullChunkData, m_rawData, and m_writeCache.
|
private |
|
virtual |
This will add the given caching algorithm to the list of attempted caching algorithms.
The algorithms are tried in the opposite order that they were added - the first algorithm added is the last algorithm tried.
This method takes ownership of algorithm.
| algorithm | The caching algorithm to add to the Cube for I/O |
Reimplemented from Isis::ImageIoHandler.
Definition at line 355 of file CubeIoHandler.cpp.
References m_cachingAlgorithms.
| int Isis::CubeIoHandler::bandCount | ( | ) | const |
Definition at line 422 of file CubeIoHandler.cpp.
References m_numBands.
Referenced by findCubeChunks().
|
private |
This blocks (doesn't return) until the number of active runnables in the thread pool goes to 0.
This uses the m_writeThreadMutex, because the thread pool doesn't delete threads immediately, so runnables might actually still exist but are inconsequential/being deleted already.
Definition at line 627 of file CubeIoHandler.cpp.
References m_ioThreadPool, and Isis::ImageIoHandler::m_writeThreadMutex.
Referenced by flushWriteCache().
This is used for sorting buffers into the most efficient write order.
| lhs | The left hand side of the '<' operator |
| rhs | The right hand side of the '<' operator |
Definition at line 641 of file CubeIoHandler.cpp.
References Isis::Buffer::Band(), Isis::Buffer::BandDimension(), Isis::Area3D::intersect(), Isis::Area3D::isValid(), Isis::Buffer::Line(), Isis::Buffer::LineDimension(), Isis::Displacement::Pixels, Isis::Distance::Pixels, Isis::Buffer::Sample(), and Isis::Buffer::SampleDimension().
|
virtual |
Free all cube chunks (cached cube data) from memory and write them to disk.
Child destructors need to call this method.
This method should only be called otherwise when lots of cubes are in memory and the many caches cause problems with system RAM limitations.
| blockForWriteCache | This should be true unless this method is called from the write thread. |
Reimplemented from Isis::ImageIoHandler.
Definition at line 370 of file CubeIoHandler.cpp.
References CubeIoHandler(), flushWriteCache(), m_dataIsOnDiskMap, m_lastProcessByLineChunks, m_rawData, writeNullDataToDisk(), and writeRaw().
Referenced by minimizeCache(), Isis::CubeBsqHandler::~CubeBsqHandler(), and Isis::CubeTileHandler::~CubeTileHandler().
|
inherited |
Get the mutex that this IO handler is using around I/Os on the given data file.
A lock should be acquired before doing any reads/writes on the data file externally.
Definition at line 58 of file ImageIoHandler.cpp.
References m_writeThreadMutex.
|
private |
Get the cube chunks that correspond to the given cube area.
This will create and initialize the chunks if they are not already in memory.
| startSample | The starting sample of the cube data |
| numSamples | The number of samples of cube data |
| startLine | The starting line of the cube data |
| numLines | The number of lines of cube data |
| startBand | The starting band of the cube data |
| numBands | The number of bands of cube data |
| Cube |
|---|
| -----— |
| | | <---------— Requested Area | | | | | -----— |
But our cube is split into "chunks" which may not line up to what was requested:
| | | || |------------------—|| | -----— | || |—|---—|-------—|| (User requested area intersected with | | | | | || cube chunks) |------------------—||
| 1 | 2 | 3 || |------------------—|| | 4 | 5 | 6 || |------------------—|| (User requested area intersected with | 7 | 8 | 9 || cube chunks) |------------------—||
(In this case we want chunks 4, 5, 7, and 8)
So we want to find which cube chunks line up to the requested area. To do this, we find which chunk intersects the top left of the requested cube area. That chunk is a given that we need it. Then we search in the positive X direction for more chunks that intersect the user-requested area until we no longer do. Then we know all of the chunks that intersect a whole y-range of the requested area. We shrink the requested area (remove the found y-range) and repeat.
Definition at line 690 of file CubeIoHandler.cpp.
References bandCount(), getChunk(), getChunkCountInLineDimension(), getChunkCountInSampleDimension(), lineCount(), m_bandsInChunk, m_linesInChunk, m_samplesInChunk, Isis::ImageIoHandler::m_virtualBands, and sampleCount().
Referenced by read(), and synchronousWrite().
|
private |
Find the intersection between the buffer area and the cube chunk.
This accounts for virtual bands when considering the buffer and may return an area where all of the bands don't actually intersect due to virtual bands. This is implemented the way it is for performance reasons - so much as returning an Area3D or a QList<int> caused undesirable slowdowns.
| cube1 | The cube chunk to intersect |
| cube2 | The buffer (in virtual band space) to intersect |
| startX | (output) The leftmost sample position (inclusive) |
| startY | (output) The topmost line position (inclusive) |
| startZ | (output) The frontmost band position (inclusive) |
| endX | (output) The rightmost sample position (inclusive) |
| endY | (output) The bottommost line position (inclusive) |
| endZ | (output) The backmost band position (inclusive) |
Definition at line 835 of file CubeIoHandler.cpp.
References Isis::Buffer::Band(), Isis::RawCubeChunk::bandCount(), Isis::Buffer::BandDimension(), Isis::RawCubeChunk::getStartBand(), Isis::RawCubeChunk::getStartLine(), Isis::RawCubeChunk::getStartSample(), Isis::Buffer::Line(), Isis::RawCubeChunk::lineCount(), Isis::Buffer::LineDimension(), Isis::ImageIoHandler::m_virtualBands, Isis::Buffer::Sample(), Isis::RawCubeChunk::sampleCount(), and Isis::Buffer::SampleDimension().
Referenced by writeIntoDouble(), and writeIntoRaw().
|
private |
This attempts to write the so-far-unwritten buffers from m_writeCache into the cube's RawCubeChunk cache.
This will not do anything if there is a runnable and will block if the write cache grows to be too large.
| force | Set to true to force start a flush |
Definition at line 900 of file CubeIoHandler.cpp.
References blockUntilThreadPoolEmpty(), CubeIoHandler(), m_consecutiveOverflowCount, m_idealFlushSize, m_ioThreadPool, m_lastOperationWasWrite, m_useOptimizedCubeWrite, m_writeCache, Isis::ImageIoHandler::m_writeThreadMutex, and synchronousWrite().
Referenced by clearCache(), read(), and write().
|
private |
If the chunk is dirty, then we write it to disk.
Regardless, we then free it from memory.
| chunkToFree | The chunk we're removing from memory |
Definition at line 977 of file CubeIoHandler.cpp.
References CubeIoHandler(), getChunkIndex(), Isis::RawCubeChunk::isDirty(), m_lastProcessByLineChunks, m_rawData, and writeRaw().
Referenced by minimizeCache().
| int Isis::CubeIoHandler::getBandCountInChunk | ( | ) | const |
Definition at line 430 of file CubeIoHandler.cpp.
References m_bandsInChunk.
Referenced by getChunkIndex(), getChunkPlacement(), read(), and synchronousWrite().
| BigInt Isis::CubeIoHandler::getBytesPerChunk | ( | ) | const |
Definition at line 442 of file CubeIoHandler.cpp.
References m_bandsInChunk, m_linesInChunk, m_pixelType, and m_samplesInChunk.
Referenced by getChunk(), Isis::CubeBsqHandler::getChunkStartByte(), getDataSize(), getNullChunk(), Isis::CubeTileHandler::getTileStartByte(), and minimizeCache().
|
private |
Retrieve the cached chunk at the given chunk index, if there is one.
| chunkIndex | The position of the chunk in the cube |
| allocateIfNecessary | If true, the chunk will be read into cache when necessary and this method will not return NULL. |
Definition at line 1005 of file CubeIoHandler.cpp.
References CubeIoHandler(), getBytesPerChunk(), getChunkPlacement(), getNullChunk(), m_dataIsOnDiskMap, m_rawData, readRaw(), and Isis::RawCubeChunk::setDirty().
Referenced by findCubeChunks(), read(), and synchronousWrite().
|
private |
Definition at line 1046 of file CubeIoHandler.cpp.
References getChunkCountInBandDimension(), getChunkCountInLineDimension(), and getChunkCountInSampleDimension().
Referenced by writeNullDataToDisk().
| int Isis::CubeIoHandler::getChunkCountInBandDimension | ( | ) | const |
Definition at line 452 of file CubeIoHandler.cpp.
References m_bandsInChunk, and m_numBands.
Referenced by getChunkCount(), and getDataSize().
| int Isis::CubeIoHandler::getChunkCountInLineDimension | ( | ) | const |
Definition at line 461 of file CubeIoHandler.cpp.
References m_linesInChunk, and m_numLines.
Referenced by findCubeChunks(), getChunkCount(), getChunkIndex(), getChunkPlacement(), getDataSize(), read(), and synchronousWrite().
| int Isis::CubeIoHandler::getChunkCountInSampleDimension | ( | ) | const |
Definition at line 470 of file CubeIoHandler.cpp.
References m_numSamples, and m_samplesInChunk.
Referenced by findCubeChunks(), getChunkCount(), getChunkIndex(), getChunkPlacement(), getDataSize(), read(), and synchronousWrite().
| int Isis::CubeIoHandler::getChunkIndex | ( | const RawCubeChunk & | chunk | ) | const |
Given a chunk, what's its index in the file.
Chunks are ordered from left to right, then top to bottom, then front to back (BSQ). The first chunk is at the top left of band 1 and is index 0, for example. In other words, this is going from the value of m_rawData to the key.
Chunks which sit outside of the cube entirely must not be passed into this method; the results will be wrong.
Definition at line 486 of file CubeIoHandler.cpp.
References getBandCountInChunk(), getChunkCountInLineDimension(), getChunkCountInSampleDimension(), getLineCountInChunk(), getSampleCountInChunk(), Isis::RawCubeChunk::getStartBand(), Isis::RawCubeChunk::getStartLine(), and Isis::RawCubeChunk::getStartSample().
Referenced by freeChunk(), Isis::CubeBsqHandler::getChunkStartByte(), and Isis::CubeTileHandler::getTileStartByte().
|
private |
Get the X/Y/Z (Sample,Line,Band) range of the chunk at the given index.
| chunkIndex | The chunk number in the cube file |
| startSample | (output) The leftmost sample position (inclusive) |
| startLine | (output) The topmost line position (inclusive) |
| startBand | (output) The frontmost band position (inclusive) |
| endSample | (output) The rightmost sample position (inclusive) |
| endLine | (output) The bottommost line position (inclusive) |
| endBand | (output) The backmost band position (inclusive) |
Definition at line 1064 of file CubeIoHandler.cpp.
References getBandCountInChunk(), getChunkCountInLineDimension(), getChunkCountInSampleDimension(), getLineCountInChunk(), and getSampleCountInChunk().
Referenced by getChunk(), getNullChunk(), read(), and synchronousWrite().
|
protected |
Definition at line 515 of file CubeIoHandler.cpp.
References m_dataFile.
Referenced by Isis::CubeBsqHandler::readRaw(), Isis::CubeTileHandler::readRaw(), Isis::CubeBsqHandler::writeRaw(), and Isis::CubeTileHandler::writeRaw().
|
virtual |
Implements Isis::ImageIoHandler.
Definition at line 412 of file CubeIoHandler.cpp.
References getBytesPerChunk(), getChunkCountInBandDimension(), getChunkCountInLineDimension(), and getChunkCountInSampleDimension().
Referenced by setChunkSizes().
| BigInt Isis::CubeIoHandler::getDataStartByte | ( | ) | const |
Definition at line 504 of file CubeIoHandler.cpp.
References m_startByte.
Referenced by Isis::CubeBsqHandler::getChunkStartByte(), Isis::CubeTileHandler::getTileStartByte(), and setChunkSizes().
| int Isis::CubeIoHandler::getLineCountInChunk | ( | ) | const |
Definition at line 532 of file CubeIoHandler.cpp.
References m_linesInChunk.
Referenced by getChunkIndex(), getChunkPlacement(), read(), synchronousWrite(), and Isis::CubeTileHandler::updateLabels().
|
private |
This creates a chunk filled with NULLs whose placement is at chunkIndex's position.
This is used for getting NULL-filled chunks the first time the chunk is requested.
Ownership of the return value is given to the caller.
| chunkIndex | The chunk's index which provides it's positioning. |
Definition at line 1096 of file CubeIoHandler.cpp.
References Isis::RawCubeChunk::bandCount(), getBytesPerChunk(), getChunkPlacement(), Isis::RawCubeChunk::getRawData(), Isis::RawCubeChunk::getStartBand(), Isis::RawCubeChunk::getStartLine(), Isis::RawCubeChunk::getStartSample(), Isis::RawCubeChunk::lineCount(), m_nullChunkData, Isis::RawCubeChunk::sampleCount(), Isis::Brick::SetBasePosition(), Isis::RawCubeChunk::setDirty(), Isis::RawCubeChunk::setRawData(), Isis::Buffer::size(), and writeIntoRaw().
Referenced by getChunk(), and writeNullDataToDisk().
| int Isis::CubeIoHandler::getSampleCountInChunk | ( | ) | const |
Definition at line 557 of file CubeIoHandler.cpp.
References m_samplesInChunk.
Referenced by getChunkIndex(), getChunkPlacement(), read(), synchronousWrite(), and Isis::CubeTileHandler::updateLabels().
| int Isis::CubeIoHandler::lineCount | ( | ) | const |
Definition at line 524 of file CubeIoHandler.cpp.
References m_numLines.
Referenced by Isis::CubeBsqHandler::CubeBsqHandler(), and findCubeChunks().
|
private |
Apply the caching algorithms and get rid of excess cube data in memory.
This is intended to be called after every IO operation.
| justUsed | The cube chunks that were used in the IO operation that is calling this method. |
| justRequested | The buffer that was used in the IO operation that is calling this method. |
Definition at line 1148 of file CubeIoHandler.cpp.
References Isis::CubeCachingAlgorithm::CacheResult::algorithmUnderstoodData(), clearCache(), freeChunk(), getBytesPerChunk(), Isis::CubeCachingAlgorithm::CacheResult::getChunksToFree(), m_cachingAlgorithms, m_rawData, and Isis::CubeCachingAlgorithm::recommendChunksToFree().
Referenced by read(), and synchronousWrite().
|
private |
Disallow assignments of this object.
| other | The CubeIoHandler on the right-hand side of the assignment that we are copying into *this. |
References CubeIoHandler().
| PixelType Isis::CubeIoHandler::pixelType | ( | ) | const |
Definition at line 540 of file CubeIoHandler.cpp.
References m_pixelType.
Referenced by Isis::CubeBsqHandler::CubeBsqHandler().
|
virtual |
Read cube data from disk into the buffer.
This is not const because it caches the read cube chunks from the disk.
| bufferToFill | The buffer to populate with cube data. |
Implements Isis::ImageIoHandler.
Definition at line 217 of file CubeIoHandler.cpp.
References Isis::Buffer::Band(), Isis::Buffer::BandDimension(), findCubeChunks(), flushWriteCache(), getBandCountInChunk(), getChunk(), getChunkCountInLineDimension(), getChunkCountInSampleDimension(), getChunkPlacement(), getLineCountInChunk(), getSampleCountInChunk(), Isis::Buffer::Line(), Isis::Buffer::LineDimension(), m_bandsInChunk, m_ioThreadPool, m_lastOperationWasWrite, m_linesInChunk, m_rawData, m_samplesInChunk, m_useOptimizedCubeWrite, Isis::ImageIoHandler::m_virtualBands, Isis::ImageIoHandler::m_writeThreadMutex, minimizeCache(), Isis::Buffer::Sample(), Isis::Buffer::SampleDimension(), Isis::Buffer::size(), and writeIntoDouble().
|
protectedpure virtual |
This needs to populate the chunkToFill with unswapped raw bytes from the disk.
| chunkToFill | The container that needs to be filled with cube data. |
Implemented in Isis::CubeBsqHandler, and Isis::CubeTileHandler.
Referenced by getChunk().
| int Isis::CubeIoHandler::sampleCount | ( | ) | const |
Definition at line 549 of file CubeIoHandler.cpp.
References m_numSamples.
Referenced by Isis::CubeBsqHandler::CubeBsqHandler(), findCubeChunks(), and synchronousWrite().
|
protected |
This should be called once from the child constructor.
This determines the chunk sizes used for the cube and often should remain constant for a particular cube (BSQ, for example, doesn't need it to be constant but Tile does). These being large can cause excessive use of disk space that stores no cube data. These being large can also drastically increase the amount of RAM consumed.
| numSamples | The chunk size in the sample dimension |
| numLines | The chunk size in the line dimension |
| numBands | The chunk size in the band dimension |
Definition at line 574 of file CubeIoHandler.cpp.
References getDataSize(), getDataStartByte(), m_bandsInChunk, m_dataFile, m_dataIsOnDiskMap, m_linesInChunk, and m_samplesInChunk.
Referenced by Isis::CubeBsqHandler::CubeBsqHandler().
|
inherited |
This changes the virtual band list.
| virtualBandList | A list where the indices are the vbands and the values are the physical bands. The values are 1-based. This can be specified as NULL, in which case the vbands are the physical bands. The virtual band list is copied (the pointer provided isn't remembered). |
Definition at line 41 of file ImageIoHandler.cpp.
References m_virtualBands.
|
private |
This method takes the given buffer and synchronously puts it into the Cube's cache.
This includes reading missing cache areas and freeing extra cache areas. This is what the non-threaded CubeIoHandler::write used to do.
| bufferToWrite | The buffer we're writing into this cube, synchronously |
Definition at line 1197 of file CubeIoHandler.cpp.
References Isis::Buffer::Band(), Isis::Buffer::BandDimension(), findCubeChunks(), getBandCountInChunk(), getChunk(), getChunkCountInLineDimension(), getChunkCountInSampleDimension(), getChunkPlacement(), getLineCountInChunk(), getSampleCountInChunk(), Isis::Buffer::Line(), Isis::Buffer::LineDimension(), m_bandsInChunk, m_lastProcessByLineChunks, m_linesInChunk, m_samplesInChunk, minimizeCache(), Isis::Buffer::Sample(), sampleCount(), Isis::Buffer::SampleDimension(), and writeIntoRaw().
Referenced by flushWriteCache(), and write().
|
pure virtual |
Function to update the labels with a Pvl object.
| labels | Pvl object to update with |
Implements Isis::ImageIoHandler.
Implemented in Isis::CubeBsqHandler, and Isis::CubeTileHandler.
|
virtual |
Write buffer data into the cube data on disk.
This could do no IO if the cube chunks required are already in memory and our caching algorithms say to not free any of the cube chunks afterwards.
| bufferToWrite | The buffer to get cube data from. |
Implements Isis::ImageIoHandler.
Definition at line 325 of file CubeIoHandler.cpp.
References flushWriteCache(), m_ioThreadPool, m_lastOperationWasWrite, m_writeCache, Isis::ImageIoHandler::m_writeThreadMutex, and synchronousWrite().
|
private |
Write the intersecting area of the chunk into the buffer.
| chunk | The data source |
| output | The data destination |
| index | int |
Definition at line 1312 of file CubeIoHandler.cpp.
References Isis::Buffer::Band(), Isis::Buffer::BandDimension(), Isis::Buffer::DoubleBuffer(), findIntersection(), Isis::RawCubeChunk::getRawData(), Isis::RawCubeChunk::getStartBand(), Isis::RawCubeChunk::getStartLine(), Isis::RawCubeChunk::getStartSample(), Isis::Buffer::Index(), Isis::RawCubeChunk::lineCount(), m_base, m_byteSwapper, m_multiplier, m_pixelType, Isis::Buffer::RawBuffer(), and Isis::RawCubeChunk::sampleCount().
Referenced by read().
|
private |
Write the intersecting area of the buffer into the chunk.
| buffer | The data source |
| output | The data destination |
| index | int |
Definition at line 1512 of file CubeIoHandler.cpp.
References Isis::Buffer::Band(), Isis::Buffer::BandDimension(), Isis::Buffer::DoubleBuffer(), findIntersection(), Isis::RawCubeChunk::getRawData(), Isis::RawCubeChunk::getStartBand(), Isis::RawCubeChunk::getStartLine(), Isis::RawCubeChunk::getStartSample(), Isis::Buffer::Index(), Isis::RawCubeChunk::lineCount(), m_base, m_byteSwapper, m_multiplier, m_pixelType, Isis::ImageIoHandler::m_virtualBands, Isis::RawCubeChunk::sampleCount(), and Isis::RawCubeChunk::setDirty().
Referenced by getNullChunk(), and synchronousWrite().
|
private |
Write all NULL cube chunks that have not yet been accessed to disk.
Definition at line 1791 of file CubeIoHandler.cpp.
References CubeIoHandler(), getChunkCount(), getNullChunk(), m_dataIsOnDiskMap, and writeRaw().
Referenced by clearCache().
|
protectedpure virtual |
This needs to write the chunkToWrite directly to disk with no modifications to the data itself.
| chunkToWrite | The container that needs to be put on disk. |
Implemented in Isis::CubeBsqHandler, and Isis::CubeTileHandler.
Referenced by clearCache(), freeChunk(), and writeNullDataToDisk().
|
private |
The number of physical bands in a cube chunk.
Definition at line 319 of file CubeIoHandler.h.
Referenced by CubeIoHandler(), findCubeChunks(), getBandCountInChunk(), getBytesPerChunk(), getChunkCountInBandDimension(), read(), setChunkSizes(), and synchronousWrite().
|
private |
The additive offset of the data on disk.
Definition at line 282 of file CubeIoHandler.h.
Referenced by CubeIoHandler(), writeIntoDouble(), and writeIntoRaw().
|
private |
The byte order (endianness) of the data on disk.
Definition at line 288 of file CubeIoHandler.h.
|
private |
A helper that swaps byte order to and from file order.
Definition at line 291 of file CubeIoHandler.h.
Referenced by CubeIoHandler(), writeIntoDouble(), writeIntoRaw(), and ~CubeIoHandler().
|
private |
The caching algorithms to use, in order of priority.
Definition at line 303 of file CubeIoHandler.h.
Referenced by addCachingAlgorithm(), CubeIoHandler(), minimizeCache(), and ~CubeIoHandler().
|
mutableprivate |
How many times the write cache has overflown in a row.
Definition at line 355 of file CubeIoHandler.h.
Referenced by CubeIoHandler(), and flushWriteCache().
|
private |
The file containing cube data.
Definition at line 269 of file CubeIoHandler.h.
Referenced by CubeIoHandler(), getDataFile(), and setChunkSizes().
|
mutableprivate |
The map from chunk index to on-disk status, all true if not allocated.
Definition at line 309 of file CubeIoHandler.h.
Referenced by clearCache(), CubeIoHandler(), getChunk(), setChunkSizes(), writeNullDataToDisk(), and ~CubeIoHandler().
|
mutableprivate |
Ideal write cache flush size.
Definition at line 352 of file CubeIoHandler.h.
Referenced by CubeIoHandler(), and flushWriteCache().
|
mutableprivate |
This contains threads for doing cube writes (and later maybe cube reads).
We're using a thread pool so that the same QThread is reused for performance reasons.
Definition at line 338 of file CubeIoHandler.h.
Referenced by blockUntilThreadPoolEmpty(), CubeIoHandler(), flushWriteCache(), read(), write(), and ~CubeIoHandler().
|
mutableprivate |
If the last operation was a write then we need to flush the cache when reading.
Keep track of this.
Definition at line 344 of file CubeIoHandler.h.
Referenced by CubeIoHandler(), flushWriteCache(), read(), and write().
|
mutableprivate |
This is an optimization for process by line.
It relies on chunks found often being the exact same between multiple reads or multiple writes.
Definition at line 325 of file CubeIoHandler.h.
Referenced by clearCache(), CubeIoHandler(), freeChunk(), synchronousWrite(), and ~CubeIoHandler().
|
private |
The number of lines in a cube chunk.
Definition at line 316 of file CubeIoHandler.h.
Referenced by CubeIoHandler(), findCubeChunks(), getBytesPerChunk(), getChunkCountInLineDimension(), getLineCountInChunk(), read(), setChunkSizes(), and synchronousWrite().
|
private |
The multiplicative factor of the data on disk.
Definition at line 285 of file CubeIoHandler.h.
Referenced by CubeIoHandler(), writeIntoDouble(), and writeIntoRaw().
|
mutableprivate |
A raw cube chunk's data when it was all NULL's. Used for speed.
Definition at line 328 of file CubeIoHandler.h.
Referenced by CubeIoHandler(), getNullChunk(), and ~CubeIoHandler().
|
private |
The number of physical bands in the cube.
Definition at line 300 of file CubeIoHandler.h.
Referenced by bandCount(), CubeIoHandler(), and getChunkCountInBandDimension().
|
private |
The number of lines in the cube.
Definition at line 297 of file CubeIoHandler.h.
Referenced by CubeIoHandler(), getChunkCountInLineDimension(), and lineCount().
|
private |
The number of samples in the cube.
Definition at line 294 of file CubeIoHandler.h.
Referenced by CubeIoHandler(), getChunkCountInSampleDimension(), and sampleCount().
|
private |
The format of each DN in the cube.
Definition at line 279 of file CubeIoHandler.h.
Referenced by CubeIoHandler(), getBytesPerChunk(), pixelType(), writeIntoDouble(), and writeIntoRaw().
|
mutableprivate |
The map from chunk index to chunk for cached data.
Definition at line 306 of file CubeIoHandler.h.
Referenced by clearCache(), CubeIoHandler(), freeChunk(), getChunk(), minimizeCache(), read(), and ~CubeIoHandler().
|
private |
The number of samples in a cube chunk.
Definition at line 313 of file CubeIoHandler.h.
Referenced by CubeIoHandler(), findCubeChunks(), getBytesPerChunk(), getChunkCountInSampleDimension(), getSampleCountInChunk(), read(), setChunkSizes(), and synchronousWrite().
|
private |
The start byte of the cube data.
This is 0-based (i.e. a value of 0 means write data into the first byte of the file). Usually the label is between 0 and m_startByte.
Definition at line 276 of file CubeIoHandler.h.
Referenced by CubeIoHandler(), and getDataStartByte().
|
private |
This is true if the Isis preference for the cube write thread is optimized.
Definition at line 349 of file CubeIoHandler.h.
Referenced by CubeIoHandler(), flushWriteCache(), and read().
|
protectedinherited |
Converts from virtual band to physical band.
Definition at line 124 of file ImageIoHandler.h.
Referenced by Isis::CubeIoHandler::findCubeChunks(), Isis::CubeIoHandler::findIntersection(), Isis::CubeIoHandler::read(), setVirtualBands(), and Isis::CubeIoHandler::writeIntoRaw().
These are the buffers we need to write to raw cube chunks.
Definition at line 331 of file CubeIoHandler.h.
Referenced by CubeIoHandler(), flushWriteCache(), write(), and ~CubeIoHandler().
|
protectedinherited |
This enables us to block while the write thread is working.
Definition at line 121 of file ImageIoHandler.h.
Referenced by Isis::CubeIoHandler::blockUntilThreadPoolEmpty(), Isis::CubeIoHandler::CubeIoHandler(), dataFileMutex(), Isis::CubeIoHandler::flushWriteCache(), Isis::CubeIoHandler::read(), and Isis::CubeIoHandler::write().