USGS

Isis 3.0 Object Programmers' Reference

 Home

Isis::CubeIoHandler Class Reference

Handles converting buffers to and from disk. More...

#include <CubeIoHandler.h>

Inheritance diagram for Isis::CubeIoHandler:
Inheritance graph
[legend]
Collaboration diagram for Isis::CubeIoHandler:
Collaboration graph
[legend]

List of all members.

Classes

class  BufferToChunkWriter
 This class is designed to handle write() asynchronously. More...

Public Member Functions

 CubeIoHandler (QFile *dataFile, const QList< int > *virtualBandList, const Pvl &label, bool alreadyOnDisk)
 Creates a new CubeIoHandler using a RegionalCachingAlgorithm.
virtual ~CubeIoHandler ()
 Cleans up all allocated memory.
void read (Buffer &bufferToFill) const
 Read cube data from disk into the buffer.
void write (const Buffer &bufferToWrite)
 Write buffer data into the cube data on disk.
void addCachingAlgorithm (CubeCachingAlgorithm *algorithm)
 This will add the given caching algorithm to the list of attempted caching algorithms.
void clearCache (bool blockForWriteCache=true) const
 Free all cube chunks (cached cube data) from memory and write them to disk.
BigInt getDataSize () const
void setVirtualBands (const QList< int > *virtualBandList)
 This changes the virtual band list.
virtual void updateLabels (Pvl &labels)
 Children should probably implement this method.
QMutex * dataFileMutex ()
 Get the mutex that this IO handler is using around I/Os on the given data file.

Protected Member Functions

int bandCount () const
int getBandCountInChunk () const
BigInt getBytesPerChunk () const
int getChunkCountInBandDimension () const
int getChunkCountInLineDimension () const
int getChunkCountInSampleDimension () const
int getChunkIndex (const RawCubeChunk &) const
 Given a chunk, what's its index in the file.
BigInt getDataStartByte () const
QFile * getDataFile ()
int lineCount () const
int getLineCountInChunk () const
PixelType pixelType () const
int sampleCount () const
int getSampleCountInChunk () const
void setChunkSizes (int numSamples, int numLines, int numBands)
 This should be called once from the child constructor.
virtual void readRaw (RawCubeChunk &chunkToFill)=0
 This needs to populate the chunkToFill with unswapped raw bytes from the disk.
virtual void writeRaw (const RawCubeChunk &chunkToWrite)=0
 This needs to write the chunkToWrite directly to disk with no modifications to the data itself.

Private Member Functions

 CubeIoHandler (const CubeIoHandler &other)
 Disallow copying of this object.
CubeIoHandler & operator= (const CubeIoHandler &other)
 Disallow assignments of this object.
void blockUntilThreadPoolEmpty () const
 This blocks (doesn't return) until the number of active runnables in the thread pool goes to 0.
QList< RawCubeChunk * > findCubeChunks (int startSample, int numSamples, int startLine, int numLines, int startBand, int numBands) const
 Get the cube chunks that correspond to the given cube area.
void findIntersection (const RawCubeChunk &cube1, const Buffer &cube2, int &startX, int &startY, int &startZ, int &endX, int &endY, int &endZ) const
 Find the intersection between the buffer area and the cube chunk.
void flushWriteCache (bool force=false) const
 This attempts to write the so-far-unwritten buffers from m_writeCache into the cube's RawCubeChunk cache.
void freeChunk (RawCubeChunk *chunkToFree) const
 If the chunk is dirty, then we write it to disk.
RawCubeChunk * getChunk (int chunkIndex, bool allocateIfNecessary) const
 Retrieve the cached chunk at the given chunk index, if there is one.
int getChunkCount () const
void getChunkPlacement (int chunkIndex, int &startSample, int &startLine, int &startBand, int &endSample, int &endLine, int &endBand) const
 Get the X/Y/Z (Sample,Line,Band) range of the chunk at the given index.
RawCubeChunk * getNullChunk (int chunkIndex) const
 This creates a chunk filled with NULLs whose placement is at chunkIndex's position.
void minimizeCache (const QList< RawCubeChunk * > &justUsed, const Buffer &justRequested) const
 Apply the caching algorithms and get rid of excess cube data in memory.
void synchronousWrite (const Buffer &bufferToWrite)
 This method takes the given buffer and synchronously puts it into the Cube's cache.
void writeIntoDouble (const RawCubeChunk &chunk, Buffer &output) const
 Write the intersecting area of the chunk into the buffer.
void writeIntoRaw (const Buffer &buffer, RawCubeChunk &output) const
 Write the intersecting area of the buffer into the chunk.
void writeNullDataToDisk () const
 Write all NULL cube chunks that have not yet been accessed to disk.

Static Private Member Functions

static bool bufferLessThan (Buffer *const &lhs, Buffer *const &rhs)
 This is used for sorting buffers into the most efficient write order.

Private Attributes

QFile * m_dataFile
 The file containing cube data.
BigInt m_startByte
 The start byte of the cube data.
PixelType m_pixelType
 The format of each DN in the cube.
double m_base
 The additive offset of the data on disk.
double m_multiplier
 The multiplicative factor of the data on disk.
ByteOrder m_byteOrder
 The byte order (endianness) of the data on disk.
EndianSwapper * m_byteSwapper
 A helper that swaps byte order to and from file order.
int m_numSamples
 The number of samples in the cube.
int m_numLines
 The number of lines in the cube.
int m_numBands
 The number of physical bands in the cube.
QList< CubeCachingAlgorithm * > * m_cachingAlgorithms
 The caching algorithms to use, in order of priority.
QMap< int, RawCubeChunk * > * m_rawData
 The map from chunk index to chunk for cached data.
QMap< int, bool > * m_dataIsOnDiskMap
 The map from chunk index to on-disk status, all true if not allocated.
QList< int > * m_virtualBands
 Converts from virtual band to physical band.
int m_samplesInChunk
 The number of samples in a cube chunk.
int m_linesInChunk
 The number of lines in a cube chunk.
int m_bandsInChunk
 The number of physical bands in a cube chunk.
QList< RawCubeChunk * > * m_lastProcessByLineChunks
 This is an optimization for process by line.
QByteArray * m_nullChunkData
 A raw cube chunk's data when it was all NULL's. Used for speed.
QPair< QMutex *, QList< Buffer * > > * m_writeCache
 These are the buffers we need to write to raw cube chunks.
QThreadPool * m_ioThreadPool
 This contains threads for doing cube writes (and later maybe cube reads).
bool m_lastOperationWasWrite
 If the last operation was a write then we need to flush the cache when reading.
bool m_useOptimizedCubeWrite
 This is true if the Isis preference for the cube write thread is optimized.
QMutex * m_writeThreadMutex
 This enables us to block while the write thread is working.
volatile int m_idealFlushSize
 Ideal write cache flush size.
int m_consecutiveOverflowCount
 How many times the write cache has overflown in a row.

Detailed Description

Handles converting buffers to and from disk.

This class handles converting buffers to and from disk. This class holds the cube chunks in memory and is capable of reading and writing them. It asks the caching algorithms to recommend cube chunks to not keep in memory. Children need to call setChunkSizes() in their constructor.

This class handles all of the virtual band conversions. This class also guarantees that unwritten cube data ends up read and written as NULLs. The default caching algorithm is a RegionalCachingAlgorithm.

Author:
2011-??-?? Jai Rideout and Steven Lambright

For internal use only.

History:
2011-06-27 Steven Lambright - Added addCachingAlgorithm()
History:
2011-07-18 Jai Rideout and Steven Lambright - Added unimplemented copy constructor and assignment operator.
History:
2011-07-18 Jai Rideout and Steven Lambright - getDataFile() is no longer const, along with readRaw() and writeRaw().
History:
2011-11-23 Jai Rideout - Added fix to findCubeChunks() and findIntersection() so that requested areas that are outside of the virtual bands of the cube will no longer fail, but instead will fill the buffer with nulls.
History:
2012-02-17 Steven Lambright - The read() method is now const - the caching should be transparent to the API. It is still important to note that you cannot call read() and write() simultaneously - Cube is guaranteeing this. Backgrounded writes are now implemented and fully supported. Timing tests show results like this: Before: User: 358.71 Kernel: 38.29 Total Elapsed: 6:49.83 After: User: 380.01 Kernel: 44.97 Total Elapsed: 6:19.96 User- user-space CPU time consumed. Kernel- kernel-space CPU time consumed. The CPU consumption and kernel time consumption goes up, but the overall run time is lower. This makes the option perfectly viable for typical multi-core desktop solutions. Turning off the cube write optimization results in equivalent run times as before implementation. These time tests were run before the adaptive cache flush size was added, which improved performance further. References #727.
History:
2012-06-06 Jeff Anderson - The read() method was modified to improve the speed for small buffer sizes as seen in ProcessByBoxcar and ProcessRubbersheet. The internal cache was always checked to be minimized which is slow for every read. Now the cache is only mimimized if it changed in size. Reference #894.

Definition at line 108 of file CubeIoHandler.h.

Constructor & Destructor Documentation

Isis::CubeIoHandler::CubeIoHandler ( QFile *  dataFile,
const QList< int > *  virtualBandList,
const Pvl &  label,
bool  alreadyOnDisk 
)

Creates a new CubeIoHandler using a RegionalCachingAlgorithm.

The chunk sizes must be set by a child in its constructor.

Parameters:
dataFile The file that contains cube data. This should be a valid, opened QFile and may not be NULL. The file should have at least read permissions.
virtualBandList A list where the indices are the vbands and the values are the physical bands. The values are 1-based. This can be specified as NULL, in which case the vbands are the physical bands. The virtual band list is copied (the pointer provided isn't remembered).
label The label which contains the "Pixels" and "Core" groups.
alreadyOnDisk True if the cube exists; false ensures all NULLs are initialized into the file before this object is destructed.

Definition at line 78 of file CubeIoHandler.cpp.

References _FILEINFO_, core(), Isis::IString::DownCase(), Isis::PvlObject::FindGroup(), Isis::PvlObject::FindKeyword(), Isis::PvlContainer::FindKeyword(), Isis::PvlObject::FindObject(), m_bandsInChunk, m_base, m_byteSwapper, m_cachingAlgorithms, m_consecutiveOverflowCount, m_dataFile, m_dataIsOnDiskMap, m_idealFlushSize, m_ioThreadPool, m_lastOperationWasWrite, m_lastProcessByLineChunks, m_linesInChunk, m_multiplier, m_nullChunkData, m_numBands, m_numLines, m_numSamples, m_pixelType, m_rawData, m_samplesInChunk, m_startByte, m_useOptimizedCubeWrite, m_virtualBands, m_writeCache, m_writeThreadMutex, Isis::PixelTypeEnumeration(), Isis::IException::Programmer, and setVirtualBands().

Isis::CubeIoHandler::~CubeIoHandler (  )  [virtual]

Cleans up all allocated memory.

Child destructors must call clearCache() because we can no longer do IO by the time this destructor is called.

Definition at line 172 of file CubeIoHandler.cpp.

References m_byteSwapper, m_cachingAlgorithms, m_dataFile, m_dataIsOnDiskMap, m_ioThreadPool, m_lastProcessByLineChunks, m_nullChunkData, m_rawData, m_virtualBands, and m_writeCache.

Isis::CubeIoHandler::CubeIoHandler ( const CubeIoHandler &  other  )  [private]

Disallow copying of this object.

Parameters:
other The object to copy.

Member Function Documentation

void Isis::CubeIoHandler::addCachingAlgorithm ( CubeCachingAlgorithm *  algorithm  ) 

This will add the given caching algorithm to the list of attempted caching algorithms.

The algorithms are tried in the opposite order that they were added - the first algorithm added is the last algorithm tried.

This method takes ownership of algorithm.

Parameters:
algorithm The caching algorithm to add to the Cube for I/O

Definition at line 372 of file CubeIoHandler.cpp.

References m_cachingAlgorithms.

Referenced by Isis::Cube::addCachingAlgorithm().

int Isis::CubeIoHandler::bandCount (  )  const [protected]
Returns:
the number of physical bands in the cube.

Definition at line 484 of file CubeIoHandler.cpp.

References m_numBands.

Referenced by findCubeChunks().

void Isis::CubeIoHandler::blockUntilThreadPoolEmpty (  )  const [private]

This blocks (doesn't return) until the number of active runnables in the thread pool goes to 0.

This uses the m_writeThreadMutex, because the thread pool doesn't delete threads immediately, so runnables might actually still exist but are inconsequential/being deleted already.

Definition at line 692 of file CubeIoHandler.cpp.

References m_ioThreadPool, and m_writeThreadMutex.

Referenced by flushWriteCache().

bool Isis::CubeIoHandler::bufferLessThan ( Buffer *const &  lhs,
Buffer *const &  rhs 
) [static, private]

This is used for sorting buffers into the most efficient write order.

Parameters:
lhs The left hand side of the '<' operator
rhs The right hand side of the '<' operator
Returns:
True if lhs is obviously earlier in a cube than rhs.

Definition at line 706 of file CubeIoHandler.cpp.

References Isis::Buffer::Band(), Isis::Buffer::BandDimension(), Isis::Area3D::intersect(), Isis::Area3D::isValid(), Isis::Buffer::Line(), Isis::Buffer::LineDimension(), Isis::Distance::Pixels, Isis::Displacement::Pixels, Isis::Buffer::Sample(), and Isis::Buffer::SampleDimension().

void Isis::CubeIoHandler::clearCache ( bool  blockForWriteCache = true  )  const

Free all cube chunks (cached cube data) from memory and write them to disk.

Child destructors need to call this method.

This method should only be called otherwise when lots of cubes are in memory and the many caches cause problems with system RAM limitations.

Parameters:
blockForWriteCache This should be true unless this method is called from the write thread.

Definition at line 387 of file CubeIoHandler.cpp.

References flushWriteCache(), m_dataIsOnDiskMap, m_lastProcessByLineChunks, m_rawData, and writeNullDataToDisk().

Referenced by Isis::Cube::clearIoCache(), minimizeCache(), Isis::CubeBsqHandler::~CubeBsqHandler(), and Isis::CubeTileHandler::~CubeTileHandler().

QMutex * Isis::CubeIoHandler::dataFileMutex (  ) 

Get the mutex that this IO handler is using around I/Os on the given data file.

A lock should be acquired before doing any reads/writes on the data file externally.

Returns:
A mutex that can guarantee exclusive access to the data file

Definition at line 477 of file CubeIoHandler.cpp.

References m_writeThreadMutex.

Referenced by Isis::Cube::read(), Isis::Cube::write(), and Isis::Cube::writeLabels().

QList< RawCubeChunk * > Isis::CubeIoHandler::findCubeChunks ( int  startSample,
int  numSamples,
int  startLine,
int  numLines,
int  startBand,
int  numBands 
) const [private]

Get the cube chunks that correspond to the given cube area.

This will create and initialize the chunks if they are not already in memory.

Parameters:
startSample The starting sample of the cube data
numSamples The number of samples of cube data
startLine The starting line of the cube data
numLines The number of lines of cube data
startBand The starting band of the cube data
numBands The number of bands of cube data
Returns:
The cube chunks that correspond to the given cube area

The user has requested a slice of a cube (a rectangle): ----------------------- | Cube | | | | -------- | | | | <------------ Requested Area | | | | | -------- | | | -----------------------

But our cube is split into "chunks" which may not line up to what was requested:

------------------------ | | | || |---------------------|| | -------- | || |---|------|----------|| (User requested area intersected with | | | | | || cube chunks) |---------------------|| | | | || ------------------------

------------------------ | 1 | 2 | 3 || |---------------------|| | 4 | 5 | 6 || |---------------------|| (User requested area intersected with | 7 | 8 | 9 || cube chunks) |---------------------|| | 10 | 11 | 12 || ------------------------

(In this case we want chunks 4, 5, 7, and 8)

So we want to find which cube chunks line up to the requested area. To do this, we find which chunk intersects the top left of the requested cube area. That chunk is a given that we need it. Then we search in the positive X direction for more chunks that intersect the user-requested area until we no longer do. Then we know all of the chunks that intersect a whole y-range of the requested area. We shrink the requested area (remove the found y-range) and repeat.

Definition at line 755 of file CubeIoHandler.cpp.

References band, bandCount(), getChunk(), getChunkCountInLineDimension(), getChunkCountInSampleDimension(), lastBand, lineCount(), m_bandsInChunk, m_linesInChunk, m_samplesInChunk, m_virtualBands, min(), results(), and sampleCount().

Referenced by read(), and synchronousWrite().

void Isis::CubeIoHandler::findIntersection ( const RawCubeChunk &  cube1,
const Buffer &  cube2,
int &  startX,
int &  startY,
int &  startZ,
int &  endX,
int &  endY,
int &  endZ 
) const [private]

Find the intersection between the buffer area and the cube chunk.

This accounts for virtual bands when considering the buffer and may return an area where all of the bands don't actually intersect due to virtual bands. This is implemented the way it is for performance reasons - so much as returning an Area3D or a QList<int> caused undesirable slowdowns.

Parameters:
cube1 The cube chunk to intersect
cube2 The buffer (in virtual band space) to intersect
startX (output) The leftmost sample position (inclusive)
startY (output) The topmost line position (inclusive)
startZ (output) The frontmost band position (inclusive)
endX (output) The rightmost sample position (inclusive)
endY (output) The bottommost line position (inclusive)
endZ (output) The backmost band position (inclusive)

Definition at line 896 of file CubeIoHandler.cpp.

References Isis::Buffer::Band(), Isis::RawCubeChunk::bandCount(), Isis::Buffer::BandDimension(), Isis::RawCubeChunk::getStartBand(), Isis::RawCubeChunk::getStartLine(), Isis::RawCubeChunk::getStartSample(), Isis::Buffer::Line(), Isis::RawCubeChunk::lineCount(), Isis::Buffer::LineDimension(), m_virtualBands, min(), Isis::Buffer::Sample(), Isis::RawCubeChunk::sampleCount(), and Isis::Buffer::SampleDimension().

Referenced by writeIntoDouble(), and writeIntoRaw().

void Isis::CubeIoHandler::flushWriteCache ( bool  force = false  )  const [private]

This attempts to write the so-far-unwritten buffers from m_writeCache into the cube's RawCubeChunk cache.

This will not do anything if there is a runnable and will block if the write cache grows to be too large.

Parameters:
optimize Set to true for best performance, false for just get it done. When optimize is true, the cache may not be flushed even though it is flushable.

Definition at line 963 of file CubeIoHandler.cpp.

References blockUntilThreadPoolEmpty(), m_consecutiveOverflowCount, m_idealFlushSize, m_ioThreadPool, m_lastOperationWasWrite, m_useOptimizedCubeWrite, m_writeCache, m_writeThreadMutex, and synchronousWrite().

Referenced by clearCache(), read(), and write().

void Isis::CubeIoHandler::freeChunk ( RawCubeChunk *  chunkToFree  )  const [private]

If the chunk is dirty, then we write it to disk.

Regardless, we then free it from memory.

Parameters:
chunkToFree The chunk we're removing from memory

Definition at line 1040 of file CubeIoHandler.cpp.

References getChunkIndex(), Isis::RawCubeChunk::isDirty(), m_lastProcessByLineChunks, m_rawData, and writeRaw().

Referenced by minimizeCache().

int Isis::CubeIoHandler::getBandCountInChunk (  )  const [protected]
Returns:
the number of bands per chunk for this cube.

Definition at line 492 of file CubeIoHandler.cpp.

References m_bandsInChunk.

Referenced by getChunkIndex(), getChunkPlacement(), read(), and synchronousWrite().

BigInt Isis::CubeIoHandler::getBytesPerChunk (  )  const [protected]
Returns:
the byte size of each chunk in the cube. Currently they must be constant size, but this is planned to be changed at some point in time.
Number of bytes in a cube chunk

Definition at line 504 of file CubeIoHandler.cpp.

References m_bandsInChunk, m_linesInChunk, m_pixelType, m_samplesInChunk, and Isis::SizeOf().

Referenced by getChunk(), Isis::CubeBsqHandler::getChunkStartByte(), getDataSize(), getNullChunk(), Isis::CubeTileHandler::getTileStartByte(), and minimizeCache().

RawCubeChunk * Isis::CubeIoHandler::getChunk ( int  chunkIndex,
bool  allocateIfNecessary 
) const [private]

Retrieve the cached chunk at the given chunk index, if there is one.

Parameters:
chunkIndex The position of the chunk in the cube
allocateIfNecessary If true, the chunk will be read into cache when necessary and this method will not return NULL.
Returns:
NULL if data is not cached, otherwise the cube file data

Definition at line 1068 of file CubeIoHandler.cpp.

References getBytesPerChunk(), getChunkPlacement(), getNullChunk(), m_dataIsOnDiskMap, m_rawData, and Isis::RawCubeChunk::setDirty().

Referenced by findCubeChunks(), read(), and synchronousWrite().

int Isis::CubeIoHandler::getChunkCount (  )  const [private]
Returns:
The number of chunks that are required to encapsulate all of the cube data

Definition at line 1109 of file CubeIoHandler.cpp.

References getChunkCountInBandDimension(), getChunkCountInLineDimension(), and getChunkCountInSampleDimension().

Referenced by writeNullDataToDisk().

int Isis::CubeIoHandler::getChunkCountInBandDimension (  )  const [protected]
Returns:
the total number of chunks in the band (Z) dimension. This is always enough to contain every band in the cube.

Definition at line 514 of file CubeIoHandler.cpp.

References m_bandsInChunk, and m_numBands.

Referenced by getChunkCount(), and getDataSize().

int Isis::CubeIoHandler::getChunkCountInLineDimension (  )  const [protected]
Returns:
the total number of chunks in the line (Y) dimension. This is always enough to contain every line in the cube.

Definition at line 523 of file CubeIoHandler.cpp.

References m_linesInChunk, and m_numLines.

Referenced by findCubeChunks(), getChunkCount(), getChunkIndex(), getChunkPlacement(), getDataSize(), read(), and synchronousWrite().

int Isis::CubeIoHandler::getChunkCountInSampleDimension (  )  const [protected]
Returns:
the total number of chunks in the sample (X) dimension. This is always enough to contain every sample in the cube.

Definition at line 532 of file CubeIoHandler.cpp.

References m_numSamples, and m_samplesInChunk.

Referenced by findCubeChunks(), getChunkCount(), getChunkIndex(), getChunkPlacement(), getDataSize(), read(), and synchronousWrite().

int Isis::CubeIoHandler::getChunkIndex ( const RawCubeChunk &  chunk  )  const [protected]

Given a chunk, what's its index in the file.

Chunks are ordered from left to right, then top to bottom, then front to back (BSQ). The first chunk is at the top left of band 1 and is index 0, for example. In other words, this is going from the value of m_rawData to the key.

Chunks which sit outside of the cube entirely must not be passed into this method; the results will be wrong.

Returns:
The chunk's index into the file

Definition at line 548 of file CubeIoHandler.cpp.

References getBandCountInChunk(), getChunkCountInLineDimension(), getChunkCountInSampleDimension(), getLineCountInChunk(), getSampleCountInChunk(), Isis::RawCubeChunk::getStartBand(), Isis::RawCubeChunk::getStartLine(), and Isis::RawCubeChunk::getStartSample().

Referenced by freeChunk(), Isis::CubeBsqHandler::getChunkStartByte(), and Isis::CubeTileHandler::getTileStartByte().

void Isis::CubeIoHandler::getChunkPlacement ( int  chunkIndex,
int &  startSample,
int &  startLine,
int &  startBand,
int &  endSample,
int &  endLine,
int &  endBand 
) const [private]

Get the X/Y/Z (Sample,Line,Band) range of the chunk at the given index.

Parameters:
chunkIndex The chunk number in the cube file
startSample (output) The leftmost sample position (inclusive)
startLine (output) The topmost line position (inclusive)
startBand (output) The frontmost band position (inclusive)
endSample (output) The rightmost sample position (inclusive)
endLine (output) The bottommost line position (inclusive)
endBand (output) The backmost band position (inclusive)

Definition at line 1127 of file CubeIoHandler.cpp.

References getBandCountInChunk(), getChunkCountInLineDimension(), getChunkCountInSampleDimension(), getLineCountInChunk(), and getSampleCountInChunk().

Referenced by getChunk(), getNullChunk(), read(), and synchronousWrite().

QFile * Isis::CubeIoHandler::getDataFile (  )  [protected]
Returns:
the QFile containing cube data. This is what should be read from and written to.
The data file for I/O

Definition at line 580 of file CubeIoHandler.cpp.

References m_dataFile.

Referenced by Isis::CubeTileHandler::readRaw(), Isis::CubeBsqHandler::readRaw(), Isis::CubeTileHandler::writeRaw(), and Isis::CubeBsqHandler::writeRaw().

BigInt Isis::CubeIoHandler::getDataSize (  )  const
Returns:
the number of bytes that the cube DNs will take up. This includes padding caused by the cube chunks not aligning with the cube dimensions.

Definition at line 429 of file CubeIoHandler.cpp.

References getBytesPerChunk(), getChunkCountInBandDimension(), getChunkCountInLineDimension(), and getChunkCountInSampleDimension().

Referenced by setChunkSizes(), and Isis::Cube::write().

BigInt Isis::CubeIoHandler::getDataStartByte (  )  const [protected]
Returns:
the byte offset to the beginning of the cube data.

Definition at line 569 of file CubeIoHandler.cpp.

References m_startByte.

Referenced by Isis::CubeBsqHandler::getChunkStartByte(), Isis::CubeTileHandler::getTileStartByte(), and setChunkSizes().

int Isis::CubeIoHandler::getLineCountInChunk (  )  const [protected]
Returns:
the number of lines in each chunk of the cube.

Definition at line 597 of file CubeIoHandler.cpp.

References m_linesInChunk.

Referenced by getChunkIndex(), getChunkPlacement(), read(), synchronousWrite(), and Isis::CubeTileHandler::updateLabels().

RawCubeChunk * Isis::CubeIoHandler::getNullChunk ( int  chunkIndex  )  const [private]

This creates a chunk filled with NULLs whose placement is at chunkIndex's position.

This is used for getting NULL-filled chunks the first time the chunk is requested.

Ownership of the return value is given to the caller.

Parameters:
chunkIndex The chunk's index which provides it's positioning.
Returns:
A chunk filled with nulls at the chunkIndex position

Definition at line 1159 of file CubeIoHandler.cpp.

References Isis::RawCubeChunk::bandCount(), getBytesPerChunk(), getChunkPlacement(), Isis::RawCubeChunk::getRawData(), Isis::RawCubeChunk::getStartBand(), Isis::RawCubeChunk::getStartLine(), Isis::RawCubeChunk::getStartSample(), Isis::RawCubeChunk::lineCount(), m_nullChunkData, Isis::Null, Isis::RawCubeChunk::sampleCount(), Isis::RawCubeChunk::setDirty(), Isis::RawCubeChunk::setRawData(), and writeIntoRaw().

Referenced by getChunk(), and writeNullDataToDisk().

int Isis::CubeIoHandler::getSampleCountInChunk (  )  const [protected]
Returns:
the number of samples in each chunk of the cube.

Definition at line 622 of file CubeIoHandler.cpp.

References m_samplesInChunk.

Referenced by getChunkIndex(), getChunkPlacement(), read(), synchronousWrite(), and Isis::CubeTileHandler::updateLabels().

int Isis::CubeIoHandler::lineCount (  )  const [protected]
Returns:
the number of lines in the cube. This does not include lines created by the chunk overflowing the line dimension.

Definition at line 589 of file CubeIoHandler.cpp.

References m_numLines.

Referenced by Isis::CubeBsqHandler::CubeBsqHandler(), Isis::CubeTileHandler::CubeTileHandler(), and findCubeChunks().

void Isis::CubeIoHandler::minimizeCache ( const QList< RawCubeChunk * > &  justUsed,
const Buffer &  justRequested 
) const [private]

Apply the caching algorithms and get rid of excess cube data in memory.

This is intended to be called after every IO operation.

Parameters:
justUsed The cube chunks that were used in the IO operation that is calling this method.
justRequested The buffer that was used in the IO operation that is calling this method.

Definition at line 1212 of file CubeIoHandler.cpp.

References Isis::CubeCachingAlgorithm::CacheResult::algorithmUnderstoodData(), clearCache(), freeChunk(), getBytesPerChunk(), Isis::CubeCachingAlgorithm::CacheResult::getChunksToFree(), m_cachingAlgorithms, m_rawData, Isis::CubeCachingAlgorithm::recommendChunksToFree(), and size.

Referenced by read(), and synchronousWrite().

CubeIoHandler& Isis::CubeIoHandler::operator= ( const CubeIoHandler &  other  )  [private]

Disallow assignments of this object.

Parameters:
other The CubeIoHandler on the right-hand side of the assignment that we are copying into *this.
Returns:
A reference to *this.

Reimplemented in Isis::CubeBsqHandler, and Isis::CubeTileHandler.

PixelType Isis::CubeIoHandler::pixelType (  )  const [protected]
Returns:
the physical cube DN format.

Definition at line 605 of file CubeIoHandler.cpp.

References m_pixelType.

Referenced by Isis::CubeTileHandler::CubeTileHandler().

void Isis::CubeIoHandler::read ( Buffer &  bufferToFill  )  const

Read cube data from disk into the buffer.

This is not const because it caches the read cube chunks from the disk.

Parameters:
bufferToFill The buffer to populate with cube data.

Definition at line 244 of file CubeIoHandler.cpp.

References Isis::Buffer::Band(), Isis::Buffer::BandDimension(), findCubeChunks(), flushWriteCache(), getBandCountInChunk(), getChunk(), getChunkCountInLineDimension(), getChunkCountInSampleDimension(), getChunkPlacement(), getLineCountInChunk(), getSampleCountInChunk(), Isis::Buffer::Line(), Isis::Buffer::LineDimension(), m_bandsInChunk, m_ioThreadPool, m_lastOperationWasWrite, m_linesInChunk, m_rawData, m_samplesInChunk, m_useOptimizedCubeWrite, m_writeThreadMutex, minimizeCache(), Isis::Null, Isis::Buffer::Sample(), Isis::Buffer::SampleDimension(), Isis::Buffer::size(), and writeIntoDouble().

Referenced by Isis::Cube::read().

virtual void Isis::CubeIoHandler::readRaw ( RawCubeChunk &  chunkToFill  )  [protected, pure virtual]

This needs to populate the chunkToFill with unswapped raw bytes from the disk.

See also:
CubeTileHandler::readRaw()
Parameters:
chunkToFill The container that needs to be filled with cube data.

Implemented in Isis::CubeBsqHandler, and Isis::CubeTileHandler.

int Isis::CubeIoHandler::sampleCount (  )  const [protected]
Returns:
the number of samples in the cube. This does not include samples created by the chunk overflowing the sample dimension.

Definition at line 614 of file CubeIoHandler.cpp.

References m_numSamples.

Referenced by Isis::CubeBsqHandler::CubeBsqHandler(), Isis::CubeTileHandler::CubeTileHandler(), findCubeChunks(), and synchronousWrite().

void Isis::CubeIoHandler::setChunkSizes ( int  numSamples,
int  numLines,
int  numBands 
) [protected]

This should be called once from the child constructor.

This determines the chunk sizes used for the cube and often should remain constant for a particular cube (BSQ, for example, doesn't need it to be constant but Tile does). These being large can cause excessive use of disk space that stores no cube data. These being large can also drastically increase the amount of RAM consumed.

Parameters:
numSamples The chunk size in the sample dimension
numLines The chunk size in the line dimension
numBands The chunk size in the band dimension

Definition at line 639 of file CubeIoHandler.cpp.

References _FILEINFO_, getDataSize(), getDataStartByte(), m_bandsInChunk, m_dataFile, m_dataIsOnDiskMap, m_linesInChunk, m_samplesInChunk, and Isis::IException::Programmer.

Referenced by Isis::CubeBsqHandler::CubeBsqHandler(), and Isis::CubeTileHandler::CubeTileHandler().

void Isis::CubeIoHandler::setVirtualBands ( const QList< int > *  virtualBandList  ) 

This changes the virtual band list.

Parameters:
virtualBandList A list where the indices are the vbands and the values are the physical bands. The values are 1-based. This can be specified as NULL, in which case the vbands are the physical bands. The virtual band list is copied (the pointer provided isn't remembered).

Definition at line 446 of file CubeIoHandler.cpp.

References m_virtualBands.

Referenced by CubeIoHandler(), and Isis::Cube::setVirtualBands().

void Isis::CubeIoHandler::synchronousWrite ( const Buffer &  bufferToWrite  )  [private]

This method takes the given buffer and synchronously puts it into the Cube's cache.

This includes reading missing cache areas and freeing extra cache areas. This is what the non-threaded CubeIoHandler::write used to do.

Parameters:
bufferToWrite The buffer we're writing into this cube, synchronously

Definition at line 1261 of file CubeIoHandler.cpp.

References Isis::Buffer::Band(), Isis::Buffer::BandDimension(), findCubeChunks(), getBandCountInChunk(), getChunk(), getChunkCountInLineDimension(), getChunkCountInSampleDimension(), getChunkPlacement(), getLineCountInChunk(), getSampleCountInChunk(), Isis::Buffer::Line(), Isis::Buffer::LineDimension(), m_bandsInChunk, m_lastProcessByLineChunks, m_linesInChunk, m_samplesInChunk, minimizeCache(), Isis::Buffer::Sample(), sampleCount(), Isis::Buffer::SampleDimension(), and writeIntoRaw().

Referenced by flushWriteCache(), Isis::CubeIoHandler::BufferToChunkWriter::run(), and write().

void Isis::CubeIoHandler::updateLabels ( Pvl &  labels  )  [virtual]

Children should probably implement this method.

This is called to allow the IO Handling algorithm to change the labels to include things like TileSamples/TileLines for example.

Parameters:
labels The PVL cube label to be updated to reflect information that the child IO handlers need to properly re-read the cube.

Reimplemented in Isis::CubeBsqHandler, and Isis::CubeTileHandler.

Definition at line 466 of file CubeIoHandler.cpp.

Referenced by Isis::Cube::create().

void Isis::CubeIoHandler::write ( const Buffer &  bufferToWrite  ) 

Write buffer data into the cube data on disk.

This could do no IO if the cube chunks required are already in memory and our caching algorithms say to not free any of the cube chunks afterwards.

Parameters:
bufferToWrite The buffer to get cube data from.

Definition at line 342 of file CubeIoHandler.cpp.

References flushWriteCache(), m_ioThreadPool, m_lastOperationWasWrite, m_writeCache, m_writeThreadMutex, and synchronousWrite().

Referenced by Isis::Cube::write().

void Isis::CubeIoHandler::writeIntoDouble ( const RawCubeChunk &  chunk,
Buffer &  output 
) const [private]

Write the intersecting area of the chunk into the buffer.

Parameters:
chunk The data source
output The data destination

Definition at line 1367 of file CubeIoHandler.cpp.

References Isis::Buffer::Band(), Isis::Buffer::BandDimension(), Isis::Buffer::DoubleBuffer(), findIntersection(), Isis::EndianSwapper::Float(), Isis::RawCubeChunk::getRawData(), Isis::RawCubeChunk::getStartBand(), Isis::RawCubeChunk::getStartLine(), Isis::RawCubeChunk::getStartSample(), Isis::Buffer::Index(), Isis::RawCubeChunk::lineCount(), m_base, m_byteSwapper, m_multiplier, m_pixelType, m_virtualBands, Isis::Buffer::RawBuffer(), Isis::RawCubeChunk::sampleCount(), Isis::EndianSwapper::ShortInt(), x, y, and z.

Referenced by read().

void Isis::CubeIoHandler::writeIntoRaw ( const Buffer &  buffer,
RawCubeChunk &  output 
) const [private]

Write the intersecting area of the buffer into the chunk.

Parameters:
buffer The data source
output The data destination

Definition at line 1501 of file CubeIoHandler.cpp.

References Isis::Buffer::Band(), Isis::Buffer::BandDimension(), Isis::Buffer::DoubleBuffer(), findIntersection(), Isis::EndianSwapper::Float(), Isis::RawCubeChunk::getRawData(), Isis::RawCubeChunk::getStartBand(), Isis::RawCubeChunk::getStartLine(), Isis::RawCubeChunk::getStartSample(), Isis::Buffer::Index(), Isis::RawCubeChunk::lineCount(), m_base, m_byteSwapper, m_multiplier, m_pixelType, m_virtualBands, Isis::RawCubeChunk::sampleCount(), Isis::RawCubeChunk::setDirty(), Isis::EndianSwapper::ShortInt(), x, y, and z.

Referenced by getNullChunk(), and synchronousWrite().

void Isis::CubeIoHandler::writeNullDataToDisk (  )  const [private]

Write all NULL cube chunks that have not yet been accessed to disk.

Definition at line 1685 of file CubeIoHandler.cpp.

References _FILEINFO_, getChunkCount(), getNullChunk(), m_dataIsOnDiskMap, and Isis::IException::Programmer.

Referenced by clearCache().

virtual void Isis::CubeIoHandler::writeRaw ( const RawCubeChunk &  chunkToWrite  )  [protected, pure virtual]

This needs to write the chunkToWrite directly to disk with no modifications to the data itself.

See also:
CubeTileHandler::writeRaw()
Parameters:
chunkToWrite The container that needs to be put on disk.

Implemented in Isis::CubeBsqHandler, and Isis::CubeTileHandler.

Referenced by freeChunk().

Member Data Documentation

int Isis::CubeIoHandler::m_bandsInChunk [private]

The number of physical bands in a cube chunk.

Definition at line 317 of file CubeIoHandler.h.

Referenced by CubeIoHandler(), findCubeChunks(), getBandCountInChunk(), getBytesPerChunk(), getChunkCountInBandDimension(), read(), setChunkSizes(), and synchronousWrite().

double Isis::CubeIoHandler::m_base [private]

The additive offset of the data on disk.

Definition at line 278 of file CubeIoHandler.h.

Referenced by CubeIoHandler(), writeIntoDouble(), and writeIntoRaw().

ByteOrder Isis::CubeIoHandler::m_byteOrder [private]

The byte order (endianness) of the data on disk.

Definition at line 284 of file CubeIoHandler.h.

EndianSwapper* Isis::CubeIoHandler::m_byteSwapper [private]

A helper that swaps byte order to and from file order.

Definition at line 287 of file CubeIoHandler.h.

Referenced by CubeIoHandler(), writeIntoDouble(), writeIntoRaw(), and ~CubeIoHandler().

QList<CubeCachingAlgorithm *>* Isis::CubeIoHandler::m_cachingAlgorithms [private]

The caching algorithms to use, in order of priority.

Definition at line 299 of file CubeIoHandler.h.

Referenced by addCachingAlgorithm(), CubeIoHandler(), minimizeCache(), and ~CubeIoHandler().

int Isis::CubeIoHandler::m_consecutiveOverflowCount [mutable, private]

How many times the write cache has overflown in a row.

Definition at line 356 of file CubeIoHandler.h.

Referenced by CubeIoHandler(), and flushWriteCache().

QFile* Isis::CubeIoHandler::m_dataFile [private]

The file containing cube data.

Definition at line 265 of file CubeIoHandler.h.

Referenced by CubeIoHandler(), getDataFile(), Isis::CubeIoHandler::BufferToChunkWriter::run(), setChunkSizes(), and ~CubeIoHandler().

QMap<int, bool>* Isis::CubeIoHandler::m_dataIsOnDiskMap [mutable, private]

The map from chunk index to on-disk status, all true if not allocated.

Definition at line 305 of file CubeIoHandler.h.

Referenced by clearCache(), CubeIoHandler(), getChunk(), setChunkSizes(), writeNullDataToDisk(), and ~CubeIoHandler().

volatile int Isis::CubeIoHandler::m_idealFlushSize [mutable, private]

Ideal write cache flush size.

Definition at line 353 of file CubeIoHandler.h.

Referenced by CubeIoHandler(), flushWriteCache(), and Isis::CubeIoHandler::BufferToChunkWriter::~BufferToChunkWriter().

QThreadPool* Isis::CubeIoHandler::m_ioThreadPool [mutable, private]

This contains threads for doing cube writes (and later maybe cube reads).

We're using a thread pool so that the same QThread is reused for performance reasons.

Definition at line 336 of file CubeIoHandler.h.

Referenced by blockUntilThreadPoolEmpty(), CubeIoHandler(), flushWriteCache(), read(), write(), and ~CubeIoHandler().

bool Isis::CubeIoHandler::m_lastOperationWasWrite [mutable, private]

If the last operation was a write then we need to flush the cache when reading.

Keep track of this.

Definition at line 342 of file CubeIoHandler.h.

Referenced by CubeIoHandler(), flushWriteCache(), read(), and write().

QList<RawCubeChunk *>* Isis::CubeIoHandler::m_lastProcessByLineChunks [mutable, private]

This is an optimization for process by line.

It relies on chunks found often being the exact same between multiple reads or multiple writes.

Definition at line 323 of file CubeIoHandler.h.

Referenced by clearCache(), CubeIoHandler(), freeChunk(), synchronousWrite(), and ~CubeIoHandler().

int Isis::CubeIoHandler::m_linesInChunk [private]

The number of lines in a cube chunk.

Definition at line 314 of file CubeIoHandler.h.

Referenced by CubeIoHandler(), findCubeChunks(), getBytesPerChunk(), getChunkCountInLineDimension(), getLineCountInChunk(), read(), setChunkSizes(), and synchronousWrite().

double Isis::CubeIoHandler::m_multiplier [private]

The multiplicative factor of the data on disk.

Definition at line 281 of file CubeIoHandler.h.

Referenced by CubeIoHandler(), writeIntoDouble(), and writeIntoRaw().

QByteArray* Isis::CubeIoHandler::m_nullChunkData [mutable, private]

A raw cube chunk's data when it was all NULL's. Used for speed.

Definition at line 326 of file CubeIoHandler.h.

Referenced by CubeIoHandler(), getNullChunk(), and ~CubeIoHandler().

int Isis::CubeIoHandler::m_numBands [private]

The number of physical bands in the cube.

Definition at line 296 of file CubeIoHandler.h.

Referenced by bandCount(), CubeIoHandler(), and getChunkCountInBandDimension().

int Isis::CubeIoHandler::m_numLines [private]

The number of lines in the cube.

Definition at line 293 of file CubeIoHandler.h.

Referenced by CubeIoHandler(), getChunkCountInLineDimension(), and lineCount().

int Isis::CubeIoHandler::m_numSamples [private]

The number of samples in the cube.

Definition at line 290 of file CubeIoHandler.h.

Referenced by CubeIoHandler(), getChunkCountInSampleDimension(), and sampleCount().

PixelType Isis::CubeIoHandler::m_pixelType [private]

The format of each DN in the cube.

Definition at line 275 of file CubeIoHandler.h.

Referenced by CubeIoHandler(), getBytesPerChunk(), pixelType(), writeIntoDouble(), and writeIntoRaw().

QMap<int, RawCubeChunk *>* Isis::CubeIoHandler::m_rawData [mutable, private]

The map from chunk index to chunk for cached data.

Definition at line 302 of file CubeIoHandler.h.

Referenced by clearCache(), CubeIoHandler(), freeChunk(), getChunk(), minimizeCache(), read(), and ~CubeIoHandler().

int Isis::CubeIoHandler::m_samplesInChunk [private]

The number of samples in a cube chunk.

Definition at line 311 of file CubeIoHandler.h.

Referenced by CubeIoHandler(), findCubeChunks(), getBytesPerChunk(), getChunkCountInSampleDimension(), getSampleCountInChunk(), read(), setChunkSizes(), and synchronousWrite().

BigInt Isis::CubeIoHandler::m_startByte [private]

The start byte of the cube data.

This is 0-based (i.e. a value of 0 means write data into the first byte of the file). Usually the label is between 0 and m_startByte.

Definition at line 272 of file CubeIoHandler.h.

Referenced by CubeIoHandler(), and getDataStartByte().

bool Isis::CubeIoHandler::m_useOptimizedCubeWrite [private]

This is true if the Isis preference for the cube write thread is optimized.

Definition at line 347 of file CubeIoHandler.h.

Referenced by CubeIoHandler(), flushWriteCache(), and read().

QList<int>* Isis::CubeIoHandler::m_virtualBands [private]

Converts from virtual band to physical band.

Definition at line 308 of file CubeIoHandler.h.

Referenced by CubeIoHandler(), findCubeChunks(), findIntersection(), setVirtualBands(), writeIntoDouble(), writeIntoRaw(), and ~CubeIoHandler().

QPair< QMutex *, QList<Buffer *> >* Isis::CubeIoHandler::m_writeCache [private]

These are the buffers we need to write to raw cube chunks.

Definition at line 329 of file CubeIoHandler.h.

Referenced by CubeIoHandler(), flushWriteCache(), write(), and ~CubeIoHandler().

QMutex* Isis::CubeIoHandler::m_writeThreadMutex [private]

This enables us to block while the write thread is working.

Definition at line 350 of file CubeIoHandler.h.

Referenced by blockUntilThreadPoolEmpty(), Isis::CubeIoHandler::BufferToChunkWriter::BufferToChunkWriter(), CubeIoHandler(), dataFileMutex(), flushWriteCache(), read(), write(), and Isis::CubeIoHandler::BufferToChunkWriter::~BufferToChunkWriter().

The documentation for this class was generated from the following files: