Compare commits

..
56 Commits
Author SHA1 Message Date
Adrien Loison 6ae79b63b3 Merge pull request #67 from box/caching_strategies
Caching strategies
2015-07-14 10:58:37 -07:00
Adrien Loison 277b665dad Merge pull request #68 from box/add_coverage_abstract_reader
Improve coverage AbstractReader
2015-07-14 10:55:22 -07:00
Adrien Loison b0c7c6ca84 Improve coverage AbstractReader 2015-07-14 10:48:03 -07:00
Adrien Loison 494c506d56 Add logic to automatically select the best caching strategy
Based on the number of unique shared strings as well as the available memory amount,
one strategy will be chosen over the other.
The algorithm is based on empirical data and super safe so it may need to be tuned.
2015-07-14 02:26:01 -07:00
Adrien Loison 334f7087da Add in-memory caching strategy for shared strings
In-memory implementation using SplFixedArray
Updated code and tests to support errors when reading XML nodes (useful when reading XML files used for attacks)
Removed LIBXML_NOENT option (which DOES substitute entities...)
Added test for Quadratic Blowup attack
2015-07-13 00:29:59 -07:00
Adrien Loison 2dcb86aae9 Move shared strings caching strategy into its own component
This will help implementing different caching strategies:
- file based
- in-memory
2015-07-11 14:12:18 -07:00
Adrien Loison 3edb056286 Merge pull request #60 from Lewiscowles1986/Update-XLSX-Reader
Separated getCellValue into multiple functions
2015-07-06 10:12:35 -07:00
Lewis Cowles edc0c22009 Update XLSX.php 2015-07-06 12:10:41 +01:00
Adrien Loison 0319e578cb Update README.md 2015-07-02 16:19:55 -07:00
Lewis 1e2452934c Additional tests for Cell Types 2015-07-02 19:35:23 +01:00
Lewis 3e1793d852 Changes made to XLSX.php 2015-07-02 17:56:55 +01:00
Lewis Cowles c1b1bd0b76 Separated getCellValue into multiple functions
Author:    Lewis Cowles <lewiscowles@me.com>
Committer: Lewis <lewis@chromebook>
	modified:   src/Spout/Reader/XLSX.php
2015-07-02 17:41:00 +01:00
Adrien Loison cf239905d7 Merge pull request #61 from box/coveralls_support
Adding coveralls.io support
2015-07-01 16:20:46 -07:00
Adrien Loison 7c8f9293cc Adding coveralls.io support
Updated TravisCI config, using:
- http://docs.travis-ci.com/user/languages/php/
- https://github.com/satooshi/php-coveralls#travis-ci

Removed automatic code coverage execution when running phpunit
Added coveralls.io badge
2015-07-01 16:15:58 -07:00
Adrien Loison 503ba97e9d Merge pull request #59 from box/fix_xlsx_writer_on_windows
Fix XLSX Writer on Windows plaftorms
2015-07-01 15:29:10 -07:00
Adrien Loison b3df57d2e5 Fix XLSX Writer on Windows plaftorms
A bug was introduced, preventing Spout to create valid XLSX files on Windows.
This commits reverts the changes that introduced DIRECTORY_SEPARATOR everywhere
and fixes the original issue with the writer by normalizing paths when creating
the zipped file.
2015-07-01 15:24:58 -07:00
Adrien Loison c6ebf115fc Merge pull request #58 from box/improve_xml_security
Prevent entity loading when reading XML
2015-07-01 14:19:49 -07:00
Adrien Loison 7d922e6776 Prevent entity loading when reading XML
Added LIBXML_NOENT option when reading a XML file
libxml_disable_entity_loader(true) cannot be used because it disables
the use of XMLReader::open()... see https://bugs.php.net/bug.php?id=62577
2015-07-01 14:07:15 -07:00
Adrien Loison d4cf853270 Merge pull request #48 from box/add_support_for_more_cell_types
Add support for more cell types
2015-06-03 11:24:43 -07:00
Adrien Loison 8bac924d48 Add support for more cell types
Added proper support for booleans, dates, numbers, errors.
Added unescaping of the read string.
Fixed a bug when cells did not have any values => now returns empty string.
2015-06-03 11:19:21 -07:00
Adrien Loison 591d86cf07 Merge pull request #46 from box/remove_version_from_composer_json
Remove version from composer.json
2015-06-02 21:47:09 -07:00
Adrien Loison ba4a55292d Remove version from composer.json 2015-06-02 19:48:37 -07:00
Adrien Loison d617a50b85 Merge pull request #44 from box/bump_to_v1.0.9
Bump to version 1.0.9
2015-05-29 09:50:21 -07:00
Adrien Loison 92981fa80b Bump to version 1.0.9 2015-05-29 09:42:38 -07:00
Adrien Loison 766c733466 Update README.md
Added question about charts and formulas in the FAQ
2015-05-29 09:39:36 -07:00
Adrien Loison 46cae76546 Update README.md
Added new badges
2015-05-29 09:30:40 -07:00
Adrien Loison 03be58b84d Merge pull request #43 from box/support_files_with_formulas
Add support for files containing formulas
2015-05-29 09:08:48 -07:00
Adrien Loison b21bb86682 Add support for files containing formulas
Formulas will be skipped on reading.
The result of the formulas will be kept though.
2015-05-29 09:01:28 -07:00
Adrien Loison 8a45cec220 Merge pull request #42 from box/support_files_without_shared_strings
Support XLSX that don't have a sharedStrings.xml file
2015-05-28 18:06:09 -07:00
Adrien Loison 04d41d7c9f Support XLSX that don't have a sharedStrings.xml file 2015-05-28 17:59:30 -07:00
Adrien Loison f8f194bc8f Merge pull request #37 from box/bump_to_version_1.0.8
Bump to version 1.0.8
2015-05-12 21:13:03 -07:00
Adrien Loison 044ad11596 Bump to version 1.0.8 2015-05-12 21:09:22 -07:00
Adrien Loison 935ba1fb5e Merge pull request #36 from box/fix_zip_path_windows
Fix issue with directory separators for zip:// on Windows
2015-05-12 21:08:38 -07:00
Adrien Loison fb0175d633 Fix issue with directory separators for zip:// on Windows
Replaced "/" by DIRECTORY_SEPARATOR every time it was used with zip://
2015-05-12 20:51:57 -07:00
Adrien Loison 69190cd855 Merge pull request #34 from box/bump_version_to_1.0.7
Bump version to 1.0.7
2015-04-29 13:53:38 -07:00
Adrien Loison 13a53fb43f Bump version to 1.0.7 2015-04-29 11:56:25 -07:00
Adrien Loison a848be52de Merge pull request #33 from box/improve_code_coverage
Improve code coverage
2015-04-29 11:55:37 -07:00
Adrien Loison 52faef0168 Merge pull request #32 from box/rename_number_to_index
Rename *Number to *Index
2015-04-29 11:55:26 -07:00
Adrien Loison 3b4dfba38e Improve code coverage 2015-04-29 11:39:21 -07:00
Adrien Loison cfd3e0ffa3 Rename *Number to *Index 2015-04-29 10:48:31 -07:00
Adrien Loison c6e943041e Merge pull request #31 from box/bump_version_to_1.0.6
Bump version to 1.0.6
2015-04-29 01:10:43 -07:00
Adrien Loison 502dd54c81 Bump version to 1.0.6 2015-04-29 01:06:16 -07:00
Adrien Loison c6cfc4c80f Merge pull request #30 from box/allow_custom_sheet_name_xlsx_writer
Allow custom sheet name in the XLSX writer
2015-04-29 01:04:22 -07:00
Adrien Loison d02013c82e Allow custom sheet name in the XLSX writer
Added setter
Added test
Updated README
2015-04-29 01:01:59 -07:00
Adrien Loison 71d9c18a81 Merge pull request #29 from box/expose_sheet_xlsx_reader
Expose a Sheet object on Reader::XLSX::nextSheet()
2015-04-29 00:32:12 -07:00
Adrien Loison e9ec4e745c Expose a Sheet object on Reader::XLSX::nextSheet()
Added Sheet class for the XLSX reader that exposes basic sheet info, such as name or ID.
When retrieving the sheet data XML, added extra XML parsing to retrieve sheet data.
Added test
2015-04-29 00:27:45 -07:00
Adrien Loison e58284d27b Merge pull request #25 from box/bump_version_to_1.0.5
Bump version to 1.0.5
2015-04-16 15:22:08 -07:00
Adrien Loison 27a47f509f Bump version to 1.0.5 2015-04-16 15:05:41 -07:00
Adrien Loison 8aeb883ebc Merge pull request #23 from box/add_hhvm_and_nightly_in_travis
Add HHVM and PHP nighly in Travis
2015-04-16 15:04:07 -07:00
Adrien Loison 05c504e72c Add HHVM and PHP nighly in Travis 2015-04-16 14:56:07 -07:00
Adrien Loison abbc158877 Merge pull request #24 from box/add_improve_test_coverage
Add and improve test coverage
2015-04-16 14:55:42 -07:00
Adrien Loison 3f3461b002 Add and improve test coverage 2015-04-16 14:51:48 -07:00
Adrien Loison 2074781852 Merge pull request #22 from box/bump_version_to_1.0.4
Bump version to 1.0.4
2015-04-16 13:17:24 -07:00
Adrien Loison fd5cfc5440 Bump version to 1.0.4 2015-04-16 13:13:05 -07:00
Adrien Loison a538813827 Merge pull request #21 from box/fix_empty_shared_string_bug
Fix empty shared string bug
2015-04-16 13:08:56 -07:00
Adrien Loison 3e5ef284a5 Fix empty shared string bug
Replaced !$sharedString by $sharedString === null to avoid the case
when $sharedString = ''
2015-04-16 13:00:02 -07:00
47 changed files with 1693 additions and 290 deletions

No files matched your search

+1
View File
@@ -1,4 +1,5 @@
/tests/resources/generated
/tests/coverage
/vendor
composer.lock
/.idea
+9 -3
View File
@@ -4,9 +4,15 @@ php:
- 5.4
- 5.5
- 5.6
- nightly
- hhvm
install:
- composer self-update
- composer install --prefer-source
- composer install --no-interaction
script: phpunit
script:
- mkdir -p build/logs
- php vendor/bin/phpunit --coverage-clover build/logs/clover.xml
after_script:
- php vendor/bin/coveralls -v
+29 -3
View File
@@ -1,11 +1,15 @@
# Spout
[![Latest Stable Version](https://poser.pugx.org/box/spout/v/stable)](https://packagist.org/packages/box/spout)
[![Project Status](http://opensource.box.com/badges/active.svg)](http://opensource.box.com/badges)
[![Build Status](https://travis-ci.org/box/spout.png?branch=master)](http://travis-ci.org/box/spout)
[![Coverage Status](https://coveralls.io/repos/box/spout/badge.svg?branch=master)](https://coveralls.io/r/box/spout?branch=master)
[![Total Downloads](https://poser.pugx.org/box/spout/downloads)](https://packagist.org/packages/box/spout)
[![License](https://poser.pugx.org/box/spout/license)](https://packagist.org/packages/box/spout)
Spout is a PHP library to read and write CSV and XLSX files, in a fast and scalable way.
Contrary to other file readers or writers, it is capable of processing very large files while keeping the memory usage really low (less than 10MB).
[![Build Status](https://travis-ci.org/box/spout.png?branch=master)](http://travis-ci.org/box/spout)
[![Project Status](http://opensource.box.com/badges/active.svg)](http://opensource.box.com/badges)
## Installation
@@ -180,6 +184,23 @@ It is also possible to retrieve all the sheets currently created:
$sheets = $writer->getSheets();
```
If you rely on the sheet's name in your application, you can access it and customize it this way:
```php
// Accessing the sheet name when reading
while ($reader->hasNextSheet()) {
$sheet = $reader->nextSheet();
$sheetName = $sheet->getName();
}
// Accessing the sheet name when writing
$sheet = $writer->getCurrentSheet();
$sheetName = $sheet->getName();
// Customizing the sheet name when writing
$sheet = $writer->getCurrentSheet();
$sheetName = $sheet->setName('My custom name');
```
### Fluent interface
Because fluent interfaces are great, you can use them with Spout:
@@ -231,6 +252,11 @@ Here are a few numbers regarding the performance of Spout:
| Read XLSX (using shared strings) | 1 second | 1-2 minutes | 35-40 minutes |
| Write XLSX | 1 second | 20-25 seconds | 8-10 minutes |
#### Does Spout support charts or formulas?
No. This is a compromise to keep memory usage low. Charts and formulas requires data to be kept in memory in order to be used.
So the larger the file would be, the more memory would be consumed, preventing your code to scale well.
## Support
+2 -2
View File
@@ -2,7 +2,6 @@
"name": "box/spout",
"description": "PHP Library to read and write CSV and XLSX files, in a fast and scalable way",
"type": "library",
"version": "1.0.3",
"keywords": ["php","read","write","csv","xlsx","excel","spreadsheet","scale","memory","stream","ooxml"],
"license": "Apache-2.0",
"homepage": "https://www.github.com/box/spout",
@@ -19,7 +18,8 @@
"ext-simplexml": "*"
},
"require-dev": {
"phpunit/phpunit": ">=3.7"
"phpunit/phpunit": ">=3.7",
"satooshi/php-coveralls": "^0.6.1"
},
"autoload": {
"psr-4": {
+6
View File
@@ -14,4 +14,10 @@
</testsuite>
</testsuites>
<filter>
<whitelist>
<directory suffix=".php">src/</directory>
</whitelist>
</filter>
</phpunit>
+4
View File
@@ -13,6 +13,8 @@ class CSV implements EscaperInterface
/**
* Escapes the given string to make it compatible with CSV
*
* @codeCoverageIgnore
*
* @param string $string The string to escape
* @return string The escaped string
*/
@@ -24,6 +26,8 @@ class CSV implements EscaperInterface
/**
* Unescapes the given string to make it compatible with CSV
*
* @codeCoverageIgnore
*
* @param string $string The string to unescape
* @return string The unescaped string
*/
+2 -2
View File
@@ -36,7 +36,7 @@ class FileSystemHelper
{
$this->throwIfOperationNotInBaseFolder($parentFolderPath);
$folderPath = $parentFolderPath . DIRECTORY_SEPARATOR . $folderName;
$folderPath = $parentFolderPath . '/' . $folderName;
$wasCreationSuccessful = mkdir($folderPath, 0777, true);
if (!$wasCreationSuccessful) {
@@ -60,7 +60,7 @@ class FileSystemHelper
{
$this->throwIfOperationNotInBaseFolder($parentFolderPath);
$filePath = $parentFolderPath . DIRECTORY_SEPARATOR . $fileName;
$filePath = $parentFolderPath . '/' . $fileName;
$wasCreationSuccessful = file_put_contents($filePath, $fileContents);
if (!$wasCreationSuccessful) {
@@ -6,6 +6,8 @@ namespace Box\Spout\Common\Helper;
* Class GlobalFunctionsHelper
* This class wraps global functions to facilitate testing
*
* @codeCoverageIgnore
*
* @package Box\Spout\Common\Helper
*/
class GlobalFunctionsHelper
@@ -89,6 +91,34 @@ class GlobalFunctionsHelper
return fgetcsv($handle, $length, $delimiter, $enclosure);
}
/**
* Wrapper around global function fputcsv()
* @see fputcsv()
*
* @param resource $handle
* @param array $fields
* @param string|void $delimiter
* @param string|void $enclosure
* @return int
*/
public function fputcsv($handle, array $fields, $delimiter = null, $enclosure = null)
{
return fputcsv($handle, $fields, $delimiter, $enclosure);
}
/**
* Wrapper around global function fwrite()
* @see fwrite()
*
* @param resource $handle
* @param string $string
* @return int
*/
public function fwrite($handle, $string)
{
return fwrite($handle, $string);
}
/**
* Wrapper around global function fclose()
* @see fclose()
@@ -117,7 +147,7 @@ class GlobalFunctionsHelper
* Wrapper around global function file_exists()
* @see file_exists()
*
* @param string $filename
* @param string $fileName
* @return bool
*/
public function file_exists($fileName)
@@ -125,11 +155,23 @@ class GlobalFunctionsHelper
return file_exists($fileName);
}
/**
* Wrapper around global function file_get_contents()
* @see file_get_contents()
*
* @param string $filePath
* @return bool
*/
public function file_get_contents($filePath)
{
return file_get_contents($filePath);
}
/**
* Wrapper around global function is_readable()
* @see is_readable()
*
* @param string $filename
* @param string $fileName
* @return bool
*/
public function is_readable($fileName)
@@ -142,11 +184,12 @@ class GlobalFunctionsHelper
* @see basename()
*
* @param string $path
* @param string|void $suffix
* @return string
*/
public function basename($path)
public function basename($path, $suffix = null)
{
return basename($path);
return basename($path, $suffix);
}
/**
+1 -1
View File
@@ -14,7 +14,7 @@ use Box\Spout\Reader\Exception\EndOfFileReachedException;
*/
abstract class AbstractReader implements ReaderInterface
{
/** @var int Used to keep track of the row number */
/** @var int Used to keep track of the row index */
protected $currentRowIndex = 0;
/** @var bool Indicates whether the stream is currently open */
+1 -1
View File
@@ -25,7 +25,7 @@ class CellHelper
{
$existingIndexes = array_keys($dataArray);
$newIndexes = array_fill_keys(range(min($existingIndexes), max($existingIndexes)), $fillValue);
$newIndexes = array_fill_keys(range(0, max($existingIndexes)), $fillValue);
$dataArray += $newIndexes;
ksort($dataArray);
@@ -0,0 +1,154 @@
<?php
namespace Box\Spout\Reader\Helper\XLSX\SharedStringsCaching;
/**
* Class CachingStrategyFactory
*
* @package Box\Spout\Reader\Helper\XLSX\SharedStringsCaching
*/
class CachingStrategyFactory
{
/**
* The memory amount needed to store a string was obtained empirically from this data:
*
* ------------------------------------
* | Number of chars⁺ | Memory needed |
* ------------------------------------
* | 3,000 | 1 MB |
* | 15,000 | 2 MB |
* | 30,000 | 5 MB |
* | 75,000 | 11 MB |
* | 150,000 | 21 MB |
* | 300,000 | 43 MB |
* | 750,000 | 105 MB |
* | 1,500,000 | 210 MB |
* | 2,250,000 | 315 MB |
* | 3,000,000 | 420 MB |
* | 4,500,000 | 630 MB |
* ------------------------------------
*
* ⁺ All characters were 1 byte long
*
* This gives a linear graph where each 1-byte character requires about 150 bytes to be stored.
* Given that some characters can take up to 4 bytes, we need 600 bytes per character to be safe.
* Also, there is on average about 20 characters per cell (this is entirely empirical data...).
*
* This means that in order to store one shared string in memory, the memory amount needed is:
* => 20 * 600 ≈ 12KB
*/
const AMOUNT_MEMORY_NEEDED_PER_STRING_IN_KB = 12;
/**
* To avoid running out of memory when extracting a huge number of shared strings, they can be saved to temporary files
* instead of in memory. Then, when accessing a string, the corresponding file contents will be loaded in memory
* and the string will be quickly retrieved.
* The performance bottleneck is not when creating these temporary files, but rather when loading their content.
* Because the contents of the last loaded file stays in memory until another file needs to be loaded, it works
* best when the indexes of the shared strings are sorted in the sheet data.
* 10,000 was chosen because it creates small files that are fast to be loaded in memory.
*/
const MAX_NUM_STRINGS_PER_TEMP_FILE = 10000;
/** @var CachingStrategyFactory|null Singleton instance */
protected static $instance = null;
/**
* Private constructor for singleton
*/
private function __construct()
{
}
/**
* Returns the singleton instance of the factory
*
* @return CachingStrategyFactory
*/
public static function getInstance()
{
if (self::$instance === null) {
self::$instance = new CachingStrategyFactory();
}
return self::$instance;
}
/**
* Returns the best caching strategy, given the number of unique shared strings
* and the amount of memory available.
*
* @param int $sharedStringsUniqueCount Number of unique shared strings
* @param string|void $tempFolder Temporary folder where the temporary files to store shared strings will be stored
* @return CachingStrategyInterface The best caching strategy
*/
public function getBestCachingStrategy($sharedStringsUniqueCount, $tempFolder = null)
{
if ($this->isInMemoryStrategyUsageSafe($sharedStringsUniqueCount)) {
return new InMemoryStrategy($sharedStringsUniqueCount);
} else {
return new FileBasedStrategy($tempFolder, self::MAX_NUM_STRINGS_PER_TEMP_FILE);
}
}
/**
* Returns whether it is safe to use in-memory caching, given the number of unique shared strings
* and the amount of memory available.
*
* @param int $sharedStringsUniqueCount Number of unique shared strings
* @return bool
*/
protected function isInMemoryStrategyUsageSafe($sharedStringsUniqueCount)
{
$memoryAvailable = $this->getMemoryLimitInKB();
if ($memoryAvailable === -1) {
// if cannot get memory limit or if memory limit set as unlimited, don't trust and play safe
return ($sharedStringsUniqueCount < self::MAX_NUM_STRINGS_PER_TEMP_FILE);
} else {
$memoryNeeded = $sharedStringsUniqueCount * self::AMOUNT_MEMORY_NEEDED_PER_STRING_IN_KB;
return ($memoryAvailable > $memoryNeeded);
}
}
/**
* Returns the PHP "memory_limit" in Kilobytes
*
* @return float
*/
protected function getMemoryLimitInKB()
{
$memoryLimitFormatted = $this->getMemoryLimitFromIni();
$memoryLimitFormatted = strtolower(trim($memoryLimitFormatted));
// No memory limit
if ($memoryLimitFormatted === '-1') {
return -1;
}
if (preg_match('/(\d+)([bkmgt])b?/', $memoryLimitFormatted, $matches)) {
$amount = intval($matches[1]);
$unit = $matches[2];
switch ($unit) {
case 'b': return ($amount / 1024);
case 'k': return $amount;
case 'm': return ($amount * 1024);
case 'g': return ($amount * 1024 * 1024);
case 't': return ($amount * 1024 * 1024 * 1024);
}
}
return -1;
}
/**
* Returns the formatted "memory_limit" value
*
* @return string
*/
protected function getMemoryLimitFromIni()
{
return ini_get('memory_limit');
}
}
@@ -0,0 +1,44 @@
<?php
namespace Box\Spout\Reader\Helper\XLSX\SharedStringsCaching;
/**
* Interface CachingStrategyInterface
*
* @package Box\Spout\Reader\Helper\XLSX\SharedStringsCaching
*/
interface CachingStrategyInterface
{
/**
* Adds the given string to the cache.
*
* @param string $sharedString The string to be added to the cache
* @param int $sharedStringIndex Index of the shared string in the sharedStrings.xml file
* @return void
*/
public function addStringForIndex($sharedString, $sharedStringIndex);
/**
* Closes the cache after the last shared string was added.
* This prevents any additional string from being added to the cache.
*
* @return void
*/
public function closeCache();
/**
* Returns the string located at the given index from the cache.
*
* @param int $sharedStringIndex Index of the shared string in the sharedStrings.xml file
* @return string The shared string at the given index
* @throws \Box\Spout\Reader\Exception\SharedStringNotFoundException If no shared string found for the given index
*/
public function getStringAtIndex($sharedStringIndex);
/**
* Destroys the cache, freeing memory and removing any created artifacts
*
* @return void
*/
public function clearCache();
}
@@ -0,0 +1,188 @@
<?php
namespace Box\Spout\Reader\Helper\XLSX\SharedStringsCaching;
use Box\Spout\Common\Helper\FileSystemHelper;
use Box\Spout\Common\Helper\GlobalFunctionsHelper;
use Box\Spout\Reader\Exception\SharedStringNotFoundException;
/**
* Class FileBasedStrategy
*
* This class implements the file-based caching strategy for shared strings.
* Shared strings are stored in small files (with a max number of strings per file).
* This strategy is slower than an in-memory strategy but is used to avoid out of memory crashes.
*
* @package Box\Spout\Reader\Helper\XLSX\SharedStringsCaching
*/
class FileBasedStrategy implements CachingStrategyInterface
{
/** Value to use to escape the line feed character ("\n") */
const ESCAPED_LINE_FEED_CHARACTER = '_x000A_';
/** @var \Box\Spout\Common\Helper\GlobalFunctionsHelper Helper to work with global functions */
protected $globalFunctionsHelper;
/** @var \Box\Spout\Common\Helper\FileSystemHelper Helper to perform file system operations */
protected $fileSystemHelper;
/**
* @var int Maximum number of strings that can be stored in one temp file
* @see CachingStrategyFactory::MAX_NUM_STRINGS_PER_TEMP_FILE
*/
protected $maxNumStringsPerTempFile;
/** @var resource Pointer to the last temp file a shared string was written to */
protected $tempFilePointer;
/**
* @var string Path of the temporary file whose contents is currently stored in memory
* @see CachingStrategyFactory::MAX_NUM_STRINGS_PER_TEMP_FILE
*/
protected $inMemoryTempFilePath;
/**
* @var string Contents of the temporary file that was last read
* @see CachingStrategyFactory::MAX_NUM_STRINGS_PER_TEMP_FILE
*/
protected $inMemoryTempFileContents;
/**
* @param string|null $tempFolder Temporary folder where the temporary files to store shared strings will be stored
* @param int $maxNumStringsPerTempFile Maximum number of strings that can be stored in one temp file
*/
public function __construct($tempFolder, $maxNumStringsPerTempFile)
{
$rootTempFolder = ($tempFolder) ?: sys_get_temp_dir();
$this->fileSystemHelper = new FileSystemHelper($rootTempFolder);
$this->tempFolder = $this->fileSystemHelper->createFolder($rootTempFolder, uniqid('sharedstrings'));
$this->maxNumStringsPerTempFile = $maxNumStringsPerTempFile;
$this->globalFunctionsHelper = new GlobalFunctionsHelper();
$this->tempFilePointer = null;
}
/**
* Adds the given string to the cache.
*
* @param string $sharedString The string to be added to the cache
* @param int $sharedStringIndex Index of the shared string in the sharedStrings.xml file
* @return void
*/
public function addStringForIndex($sharedString, $sharedStringIndex)
{
$tempFilePath = $this->getSharedStringTempFilePath($sharedStringIndex);
if (!$this->globalFunctionsHelper->file_exists($tempFilePath)) {
if ($this->tempFilePointer) {
$this->globalFunctionsHelper->fclose($this->tempFilePointer);
}
$this->tempFilePointer = $this->globalFunctionsHelper->fopen($tempFilePath, 'w');
}
// The shared string retrieval logic expects each cell data to be on one line only
// Encoding the line feed character allows to preserve this assumption
$lineFeedEncodedSharedString = $this->escapeLineFeed($sharedString);
$this->globalFunctionsHelper->fwrite($this->tempFilePointer, $lineFeedEncodedSharedString . PHP_EOL);
}
/**
* Returns the path for the temp file that should contain the string for the given index
*
* @param int $sharedStringIndex Index of the shared string in the sharedStrings.xml file
* @return string The temp file path for the given index
*/
protected function getSharedStringTempFilePath($sharedStringIndex)
{
$numTempFile = intval($sharedStringIndex / $this->maxNumStringsPerTempFile);
return $this->tempFolder . '/sharedstrings' . $numTempFile;
}
/**
* Closes the cache after the last shared string was added.
* This prevents any additional string from being added to the cache.
*
* @return void
*/
public function closeCache()
{
// close pointer to the last temp file that was written
if ($this->tempFilePointer) {
$this->globalFunctionsHelper->fclose($this->tempFilePointer);
}
}
/**
* Returns the string located at the given index from the cache.
*
* @param int $sharedStringIndex Index of the shared string in the sharedStrings.xml file
* @return string The shared string at the given index
* @throws \Box\Spout\Reader\Exception\SharedStringNotFoundException If no shared string found for the given index
*/
public function getStringAtIndex($sharedStringIndex)
{
$tempFilePath = $this->getSharedStringTempFilePath($sharedStringIndex);
$indexInFile = $sharedStringIndex % $this->maxNumStringsPerTempFile;
if (!$this->globalFunctionsHelper->file_exists($tempFilePath)) {
throw new SharedStringNotFoundException("Shared string temp file not found: $tempFilePath ; for index: $sharedStringIndex");
}
if ($this->inMemoryTempFilePath !== $tempFilePath) {
// free memory
unset($this->inMemoryTempFileContents);
$this->inMemoryTempFileContents = explode(PHP_EOL, $this->globalFunctionsHelper->file_get_contents($tempFilePath));
$this->inMemoryTempFilePath = $tempFilePath;
}
$sharedString = null;
if (array_key_exists($indexInFile, $this->inMemoryTempFileContents)) {
$escapedSharedString = $this->inMemoryTempFileContents[$indexInFile];
$sharedString = $this->unescapeLineFeed($escapedSharedString);
}
if ($sharedString === null) {
throw new SharedStringNotFoundException("Shared string not found for index: $sharedStringIndex");
}
return rtrim($sharedString, PHP_EOL);
}
/**
* Escapes the line feed characters (\n)
*
* @param string $unescapedString
* @return string
*/
private function escapeLineFeed($unescapedString)
{
return str_replace("\n", self::ESCAPED_LINE_FEED_CHARACTER, $unescapedString);
}
/**
* Unescapes the line feed characters (\n)
*
* @param string $escapedString
* @return string
*/
private function unescapeLineFeed($escapedString)
{
return str_replace(self::ESCAPED_LINE_FEED_CHARACTER, "\n", $escapedString);
}
/**
* Destroys the cache, freeing memory and removing any created artifacts
*
* @return void
*/
public function clearCache()
{
if ($this->tempFolder) {
$this->fileSystemHelper->deleteFolderRecursively($this->tempFolder);
}
}
}
@@ -0,0 +1,82 @@
<?php
namespace Box\Spout\Reader\Helper\XLSX\SharedStringsCaching;
use Box\Spout\Reader\Exception\SharedStringNotFoundException;
/**
* Class InMemoryStrategy
*
* This class implements the in-memory caching strategy for shared strings.
* This strategy is used when the number of unique strings is low, compared to the memory available.
*
* @package Box\Spout\Reader\Helper\XLSX\SharedStringsCaching
*/
class InMemoryStrategy implements CachingStrategyInterface
{
/** @var \SplFixedArray Array used to cache the shared strings */
protected $inMemoryCache;
/** @var bool Whether the cache has been closed */
protected $isCacheClosed;
/**
* @param int $sharedStringsUniqueCount Number of unique shared strings
*/
public function __construct($sharedStringsUniqueCount)
{
$this->inMemoryCache = new \SplFixedArray($sharedStringsUniqueCount);
$this->isCacheClosed = false;
}
/**
* Adds the given string to the cache.
*
* @param string $sharedString The string to be added to the cache
* @param int $sharedStringIndex Index of the shared string in the sharedStrings.xml file
* @return void
*/
public function addStringForIndex($sharedString, $sharedStringIndex)
{
if (!$this->isCacheClosed) {
$this->inMemoryCache->offsetSet($sharedStringIndex, $sharedString);
}
}
/**
* Closes the cache after the last shared string was added.
* This prevents any additional string from being added to the cache.
*
* @return void
*/
public function closeCache()
{
$this->isCacheClosed = true;
}
/**
* Returns the string located at the given index from the cache.
*
* @param int $sharedStringIndex Index of the shared string in the sharedStrings.xml file
* @return string The shared string at the given index
* @throws \Box\Spout\Reader\Exception\SharedStringNotFoundException If no shared string found for the given index
*/
public function getStringAtIndex($sharedStringIndex)
{
try {
return $this->inMemoryCache->offsetGet($sharedStringIndex);
} catch (\RuntimeException $e) {
throw new SharedStringNotFoundException("Shared string not found for index: $sharedStringIndex");
}
}
/**
* Destroys the cache, freeing memory and removing any created artifacts
*
* @return void
*/
public function clearCache()
{
unset($this->inMemoryCache);
$this->isCacheClosed = false;
}
}
@@ -3,8 +3,8 @@
namespace Box\Spout\Reader\Helper\XLSX;
use Box\Spout\Common\Exception\IOException;
use Box\Spout\Common\Helper\FileSystemHelper;
use Box\Spout\Reader\Exception\SharedStringNotFoundException;
use Box\Spout\Reader\Helper\XLSX\SharedStringsCaching\CachingStrategyFactory;
use Box\Spout\Reader\Helper\XLSX\SharedStringsCaching\CachingStrategyInterface;
/**
* Class SharedStringsHelper
@@ -20,43 +20,14 @@ class SharedStringsHelper
/** Main namespace for the sharedStrings.xml file */
const MAIN_NAMESPACE_FOR_SHARED_STRINGS_XML = 'http://schemas.openxmlformats.org/spreadsheetml/2006/main';
/**
* To avoid running out of memory when extracting the shared strings, they will be saved to temporary files
* instead of in memory. Then, when accessing a string, the corresponding file contents will be loaded in memory
* and the string will be quickly retrieved.
* The performance bottleneck is not when creating these temporary files, but rather when loading their content.
* Because the contents of the last loaded file stays in memory until another file needs to be loaded, it works
* best when the indexes of the shared strings are sorted in the sheet data.
* 10,000 was chosen because it creates small files that are fast to be loaded in memory.
*/
const MAX_NUM_STRINGS_PER_TEMP_FILE = 10000;
/** Value to use to escape the line feed character ("\n") */
const ESCAPED_LINE_FEED_CHARACTER = '_x000A_';
/** @var string Path of the XLSX file being read */
protected $filePath;
/** @var string Temporary folder where the temporary files to store shared strings will be stored */
protected $tempFolder;
/** @var \Box\Spout\Writer\Helper\XLSX\FileSystemHelper Helper to perform file system operations */
protected $fileSystemHelper;
/** @var resource Pointer to the last temp file a shared string was written to */
protected $tempFilePointer;
/**
* @var string Path of the temporary file whose contents is currently stored in memory
* @see MAX_NUM_STRINGS_PER_TEMP_FILE
*/
protected $inMemoryTempFilePath;
/**
* @var string Contents of the temporary file that was last read
* @see MAX_NUM_STRINGS_PER_TEMP_FILE
*/
protected $inMemoryTempFileContents;
/** @var CachingStrategyInterface The best caching strategy for storing shared strings */
protected $cachingStrategy;
/**
* @param string $filePath Path of the XLSX file being read
@@ -65,10 +36,25 @@ class SharedStringsHelper
public function __construct($filePath, $tempFolder = null)
{
$this->filePath = $filePath;
$this->tempFolder = $tempFolder;
}
$rootTempFolder = ($tempFolder) ?: sys_get_temp_dir();
$this->fileSystemHelper = new FileSystemHelper($rootTempFolder);
$this->tempFolder = $this->fileSystemHelper->createFolder($rootTempFolder, uniqid('sharedstrings'));
/**
* Returns whether the XLSX file contains a shared strings XML file
*
* @return bool
*/
public function hasSharedStrings()
{
$hasSharedStrings = false;
$zip = new \ZipArchive();
if ($zip->open($this->filePath) === true) {
$hasSharedStrings = ($zip->locateName(self::SHARED_STRINGS_XML_FILE_PATH) !== false);
$zip->close();
}
return $hasSharedStrings;
}
/**
@@ -90,20 +76,22 @@ class SharedStringsHelper
{
$xmlReader = new \XMLReader();
$sharedStringIndex = 0;
$this->tempFilePointer = null;
$escaper = new \Box\Spout\Common\Escaper\XLSX();
$sharedStringsFilePath = 'zip://' . $this->filePath . '#' . self::SHARED_STRINGS_XML_FILE_PATH;
$sharedStringsFilePath = $this->getSharedStringsFilePath();
if ($xmlReader->open($sharedStringsFilePath, null, LIBXML_NONET) === false) {
throw new IOException('Could not open "' . self::SHARED_STRINGS_XML_FILE_PATH . '".');
}
$sharedStringsUniqueCount = $this->getSharedStringsUniqueCount($xmlReader);
$this->cachingStrategy = $this->getBestSharedStringsCachingStrategy($sharedStringsUniqueCount);
while ($xmlReader->read() && $xmlReader->name !== 'si') {
// do nothing until a 'si' tag is reached
}
while ($xmlReader->name === 'si') {
$node = new \SimpleXMLElement($xmlReader->readOuterXml());
$node = $this->getSimpleXmlElementNodeFromXMLReader($xmlReader);
$node->registerXPathNamespace('ns', self::MAIN_NAMESPACE_FOR_SHARED_STRINGS_XML);
// removes nodes that should not be read, like the pronunciation of the Kanji characters
@@ -122,12 +110,7 @@ class SharedStringsHelper
}
$unescapedTextValue = $escaper->unescape($textValue);
// The shared string retrieval logic expects each cell data to be on one line only
// Encoding the line feed character allows to preserve this assumption
$lineFeedEncodedTextValue = $this->escapeLineFeed($unescapedTextValue);
$this->writeSharedStringToTempFile($lineFeedEncodedTextValue, $sharedStringIndex);
$this->cachingStrategy->addStringForIndex($unescapedTextValue, $sharedStringIndex);
$sharedStringIndex++;
@@ -135,14 +118,93 @@ class SharedStringsHelper
$xmlReader->next('si');
}
// close pointer to the last temp file that was written
if ($this->tempFilePointer) {
fclose($this->tempFilePointer);
}
$this->cachingStrategy->closeCache();
$xmlReader->close();
}
/**
* @return string The path to the shared strings XML file
*/
protected function getSharedStringsFilePath()
{
return 'zip://' . $this->filePath . '#' . self::SHARED_STRINGS_XML_FILE_PATH;
}
/**
* Returns the shared strings unique count, as specified in <sst> tag.
*
* @param \XMLReader $xmlReader XMLReader instance
* @return int Number of unique shared strings in the sharedStrings.xml file
* @throws \Box\Spout\Common\Exception\IOException If sharedStrings.xml is invalid and can't be read
*/
protected function getSharedStringsUniqueCount($xmlReader)
{
// Use internal errors to avoid displaying lots of warning messages in case of invalid file
// For instance, if the file is used to perform a "Billion Laughs" or "Quadratic Blowup" attacks
libxml_clear_errors();
libxml_use_internal_errors(true);
$xmlReader->next('sst');
// Iterate over the "sst" elements to get the actual "sst ELEMENT" (skips any DOCTYPE)
while ($xmlReader->name === 'sst' && $xmlReader->nodeType !== \XMLReader::ELEMENT) {
$xmlReader->read();
}
$readError = libxml_get_last_error();
if ($readError !== false) {
throw new IOException("The sharedStrings.xml file is invalid and cannot be read. [{$readError->message}]");
}
// reset the setting to display XML warnings/errors
libxml_use_internal_errors(false);
return intval($xmlReader->getAttribute('uniqueCount'));
}
/**
* Returns the best shared strings caching strategy.
*
* @param int $sharedStringsUniqueCount
* @return CachingStrategyInterface
*/
protected function getBestSharedStringsCachingStrategy($sharedStringsUniqueCount)
{
return CachingStrategyFactory::getInstance()
->getBestCachingStrategy($sharedStringsUniqueCount, $this->tempFolder);
}
/**
* Returns a SimpleXMLElement node from the current node in the given XMLReader instance.
* This is to simplify the parsing of the subtree.
*
* @param \XMLReader $xmlReader
* @return \SimpleXMLElement
* @throws \Box\Spout\Common\Exception\IOException If the current node cannot be read
*/
protected function getSimpleXmlElementNodeFromXMLReader($xmlReader)
{
// Use internal errors to avoid displaying lots of warning messages in case of error found in the XML node.
// For instance, if the file is used to perform a "Billion Laughs" or "Quadratic Blowup" attacks
libxml_clear_errors();
libxml_use_internal_errors(true);
$node = null;
try {
$node = new \SimpleXMLElement($xmlReader->readOuterXml());
} catch (\Exception $exception) {
$error = libxml_get_last_error();
libxml_use_internal_errors(false);
throw new IOException('The sharedStrings.xml file contains unreadable data [' . trim($error->message) . '].');
}
libxml_use_internal_errors(false);
return $node;
}
/**
* Removes nodes that should not be read, like the pronunciation of the Kanji characters.
* By keeping them, their text content would be added to the read string.
@@ -193,42 +255,7 @@ class SharedStringsHelper
}
/**
* Writes the given string to its associated temp file.
* A new temporary file is created when the previous one has reached its max capacity.
*
* @param string $sharedString Shared string to write to the temp file
* @param int $sharedStringIndex Index of the shared string in the sharedStrings.xml file
* @return void
*/
protected function writeSharedStringToTempFile($sharedString, $sharedStringIndex)
{
$tempFilePath = $this->getSharedStringTempFilePath($sharedStringIndex);
if (!file_exists($tempFilePath)) {
if ($this->tempFilePointer) {
fclose($this->tempFilePointer);
}
$this->tempFilePointer = fopen($tempFilePath, 'w');
}
fwrite($this->tempFilePointer, $sharedString . PHP_EOL);
}
/**
* Returns the path for the temp file that should contain the string for the given index
*
* @param int $sharedStringIndex Index of the shared string in the sharedStrings.xml file
* @return string The temp file path for the given index
*/
protected function getSharedStringTempFilePath($sharedStringIndex)
{
$numTempFile = intval($sharedStringIndex / self::MAX_NUM_STRINGS_PER_TEMP_FILE);
return $this->tempFolder . DIRECTORY_SEPARATOR . 'sharedstrings' . $numTempFile;
}
/**
* Returns the shared string at the given index.
* Because the strings have been split into different files, it looks for the value in the correct file.
* Returns the shared string at the given index, using the previously chosen caching strategy.
*
* @param int $sharedStringIndex Index of the shared string in the sharedStrings.xml file
* @return string The shared string at the given index
@@ -236,63 +263,18 @@ class SharedStringsHelper
*/
public function getStringAtIndex($sharedStringIndex)
{
$tempFilePath = $this->getSharedStringTempFilePath($sharedStringIndex);
$indexInFile = $sharedStringIndex % self::MAX_NUM_STRINGS_PER_TEMP_FILE;
if (!file_exists($tempFilePath)) {
throw new SharedStringNotFoundException("Shared string temp file not found: $tempFilePath ; for index: $sharedStringIndex");
}
if ($this->inMemoryTempFilePath !== $tempFilePath) {
// free memory
unset($this->inMemoryTempFileContents);
$this->inMemoryTempFileContents = explode(PHP_EOL, file_get_contents($tempFilePath));
$this->inMemoryTempFilePath = $tempFilePath;
}
$sharedString = null;
if (array_key_exists($indexInFile, $this->inMemoryTempFileContents)) {
$escapedSharedString = $this->inMemoryTempFileContents[$indexInFile];
$sharedString = $this->unescapeLineFeed($escapedSharedString);
}
if (!$sharedString) {
throw new SharedStringNotFoundException("Shared string not found for index: $sharedStringIndex");
}
return rtrim($sharedString, PHP_EOL);
return $this->cachingStrategy->getStringAtIndex($sharedStringIndex);
}
/**
* Escapes the line feed character (\n)
*
* @param string $unescapedString
* @return string
*/
private function escapeLineFeed($unescapedString)
{
return str_replace("\n", self::ESCAPED_LINE_FEED_CHARACTER, $unescapedString);
}
/**
* Unescapes the line feed character (\n)
*
* @param string $escapedString
* @return string
*/
private function unescapeLineFeed($escapedString)
{
return str_replace(self::ESCAPED_LINE_FEED_CHARACTER, "\n", $escapedString);
}
/**
* Deletes the created temporary folder and all its contents
* Destroys the cache, freeing memory and removing any created artifacts
*
* @return void
*/
public function cleanup()
{
$this->fileSystemHelper->deleteFolderRecursively($this->tempFolder);
if ($this->cachingStrategy) {
$this->cachingStrategy->clearCache();
}
}
}
+146 -11
View File
@@ -3,6 +3,7 @@
namespace Box\Spout\Reader\Helper\XLSX;
use Box\Spout\Reader\Internal\XLSX\Worksheet;
use Box\Spout\Reader\Sheet;
/**
* Class WorksheetHelper
@@ -12,11 +13,18 @@ use Box\Spout\Reader\Internal\XLSX\Worksheet;
*/
class WorksheetHelper
{
/** Path of Content_Types XML file inside the XLSX file */
const CONTENT_TYPES_XML_FILE_PATH = '[Content_Types].xml';
/** Extension for XML files */
const XML_EXTENSION = '.xml';
/** Main namespace for the [Content_Types].xml file */
/** Paths of XML files relative to the XLSX file root */
const CONTENT_TYPES_XML_FILE_PATH = '[Content_Types].xml';
const WORKBOOK_XML_RELS_FILE_PATH = 'xl/_rels/workbook.xml.rels';
const WORKBOOK_XML_FILE_PATH = 'xl/workbook.xml';
/** Namespaces for the XML files */
const MAIN_NAMESPACE_FOR_CONTENT_TYPES_XML = 'http://schemas.openxmlformats.org/package/2006/content-types';
const MAIN_NAMESPACE_FOR_WORKBOOK_XML_RELS = 'http://schemas.openxmlformats.org/package/2006/relationships';
const MAIN_NAMESPACE_FOR_WORKBOOK_XML = 'http://schemas.openxmlformats.org/spreadsheetml/2006/main';
/** Value of the Override attribute used in [Content_Types].xml to define worksheets */
const OVERRIDE_CONTENT_TYPES_ATTRIBUTE = 'application/vnd.openxmlformats-officedocument.spreadsheetml.worksheet+xml';
@@ -24,12 +32,23 @@ class WorksheetHelper
/** @var string Path of the XLSX file being read */
protected $filePath;
/** @var \Box\Spout\Common\Helper\GlobalFunctionsHelper Helper to work with global functions */
protected $globalFunctionsHelper;
/** @var \SimpleXMLElement XML element representing the workbook.xml.rels file */
protected $workbookXMLRelsAsXMLElement;
/** @var \SimpleXMLElement XML element representing the workbook.xml file */
protected $workbookXMLAsXMLElement;
/**
* @param string $filePath Path of the XLSX file being read
* @param \Box\Spout\Common\Helper\GlobalFunctionsHelper $globalFunctionsHelper
*/
public function __construct($filePath)
public function __construct($filePath, $globalFunctionsHelper)
{
$this->filePath = $filePath;
$this->globalFunctionsHelper = $globalFunctionsHelper;
}
/**
@@ -42,23 +61,139 @@ class WorksheetHelper
{
$worksheets = [];
$xmlContents = file_get_contents('zip://' . $this->filePath . '#' . self::CONTENT_TYPES_XML_FILE_PATH);
$contentTypes = new \SimpleXMLElement($xmlContents);
$contentTypes->registerXPathNamespace('ns', self::MAIN_NAMESPACE_FOR_CONTENT_TYPES_XML);
$contentTypesAsXMLElement = $this->getFileAsXMLElementWithNamespace(
self::CONTENT_TYPES_XML_FILE_PATH,
self::MAIN_NAMESPACE_FOR_CONTENT_TYPES_XML
);
// find all nodes defining a worksheet
$sheetNodes = $contentTypes->xpath('//ns:Override[@ContentType="' . self::OVERRIDE_CONTENT_TYPES_ATTRIBUTE . '"]');
$sheetNodes = $contentTypesAsXMLElement->xpath('//ns:Override[@ContentType="' . self::OVERRIDE_CONTENT_TYPES_ATTRIBUTE . '"]');
for ($i = 0; $i < count($sheetNodes); $i++) {
$sheetNode = $sheetNodes[$i];
$sheetDataXMLFilePath = (string) $sheetNode->attributes()->PartName;
$worksheets[] = new Worksheet($i, $sheetDataXMLFilePath);
$sheet = $this->getSheet($sheetDataXMLFilePath, $i);
$worksheets[] = new Worksheet($sheet, $i, $sheetDataXMLFilePath);
}
return $worksheets;
}
/**
* Returns an instance of a sheet, given the path of its data XML file.
* We first look at "xl/_rels/workbook.xml.rels" to find the relationship ID of the sheet.
* Then we look at "xl/worbook.xml" to find the sheet entry associated to the found ID.
* The entry contains the ID and name of the sheet.
*
* If this piece of data can't be found by parsing the different XML files, the ID will default
* to the sheet index, based on order in [Content_Types].xml. Similarly, the sheet's name will
* default to the data sheet XML file name ("xl/worksheets/sheet2.xml" => "sheet2").
*
* @param string $sheetDataXMLFilePath Path of the sheet data XML file as in [Content_Types].xml
* @param int $sheetIndexZeroBased Index of the sheet, based on order in [Content_Types].xml (zero-based)
* @return \Box\Spout\Reader\Sheet Sheet instance
*/
protected function getSheet($sheetDataXMLFilePath, $sheetIndexZeroBased)
{
$sheetId = $sheetIndexZeroBased + 1;
$sheetName = $this->getDefaultSheetName($sheetDataXMLFilePath);
/*
* In [Content_Types].xml, the path is "/xl/worksheets/sheet1.xml"
* In workbook.xml.rels, it is only "worksheets/sheet1.xml"
*/
$sheetDataXMLFilePathInWorkbookXMLRels = ltrim($sheetDataXMLFilePath, '/xl/');
// find the node associated to the given file path
$workbookXMLResElement = $this->getWorkbookXMLRelsAsXMLElement();
$relationshipNodes = $workbookXMLResElement->xpath('//ns:Relationship[@Target="' . $sheetDataXMLFilePathInWorkbookXMLRels . '"]');
if (count($relationshipNodes) === 1) {
$relationshipNode = $relationshipNodes[0];
$sheetId = (string) $relationshipNode->attributes()->Id;
$workbookXMLElement = $this->getWorkbookXMLAsXMLElement();
$sheetNodes = $workbookXMLElement->xpath('//ns:sheet[@r:id="' . $sheetId . '"]');
if (count($sheetNodes) === 1) {
$sheetNode = $sheetNodes[0];
$sheetId = (int) $sheetNode->attributes()->sheetId;
$escapedSheetName = (string) $sheetNode->attributes()->name;
$escaper = new \Box\Spout\Common\Escaper\XLSX();
$sheetName = $escaper->unescape($escapedSheetName);
}
}
return new Sheet($sheetId, $sheetIndexZeroBased, $sheetName);
}
/**
* Returns the default name of the sheet whose data is located
* at the given path.
*
* @param $sheetDataXMLFilePath
* @return string The default sheet name
*/
protected function getDefaultSheetName($sheetDataXMLFilePath)
{
return $this->globalFunctionsHelper->basename($sheetDataXMLFilePath, self::XML_EXTENSION);
}
/**
* Returns a representation of the workbook.xml.rels file, ready to be parsed.
* The returned value is cached.
*
* @return \SimpleXMLElement XML element representating the workbook.xml.rels file
*/
protected function getWorkbookXMLRelsAsXMLElement()
{
if (!$this->workbookXMLRelsAsXMLElement) {
$this->workbookXMLRelsAsXMLElement = $this->getFileAsXMLElementWithNamespace(
self::WORKBOOK_XML_RELS_FILE_PATH,
self::MAIN_NAMESPACE_FOR_WORKBOOK_XML_RELS
);
}
return $this->workbookXMLRelsAsXMLElement;
}
/**
* Returns a representation of the workbook.xml file, ready to be parsed.
* The returned value is cached.
*
* @return \SimpleXMLElement XML element representating the workbook.xml.rels file
*/
protected function getWorkbookXMLAsXMLElement()
{
if (!$this->workbookXMLAsXMLElement) {
$this->workbookXMLAsXMLElement = $this->getFileAsXMLElementWithNamespace(
self::WORKBOOK_XML_FILE_PATH,
self::MAIN_NAMESPACE_FOR_WORKBOOK_XML
);
}
return $this->workbookXMLAsXMLElement;
}
/**
* Loads the contents of the given file in an XML parser and register the given XPath namespace.
*
* @param string $xmlFilePath The path of the XML file inside the XLSX file
* @param string $mainNamespace The main XPath namespace to register
* @return \SimpleXMLElement The XML element representing the file
*/
protected function getFileAsXMLElementWithNamespace($xmlFilePath, $mainNamespace)
{
$xmlContents = $this->globalFunctionsHelper->file_get_contents('zip://' . $this->filePath . '#' . $xmlFilePath);
$xmlElement = new \SimpleXMLElement($xmlContents);
$xmlElement->registerXPathNamespace('ns', $mainNamespace);
return $xmlElement;
}
/**
* Returns whether another worksheet exists after the current worksheet.
* The order is determined by the order of appearance in the [Content_Types].xml file.
@@ -69,6 +204,6 @@ class WorksheetHelper
*/
public function hasNextWorksheet($currentWorksheet, $allWorksheets)
{
return ($currentWorksheet === null || ($currentWorksheet->getWorksheetNumber() + 1 < count($allWorksheets)));
return ($currentWorksheet === null || ($currentWorksheet->getWorksheetIndex() + 1 < count($allWorksheets)));
}
}
+23 -9
View File
@@ -10,35 +10,49 @@ namespace Box\Spout\Reader\Internal\XLSX;
*/
class Worksheet
{
/** @var int Worksheet number, based on the order of appareance in [Content_Types].xml (zero-based) */
protected $worksheetNumber;
/** @var \Box\Spout\Reader\Sheet The "external" sheet */
protected $externalSheet;
/** @var int Worksheet index, based on the order of appareance in [Content_Types].xml (zero-based) */
protected $worksheetIndex;
/** @var string Path of the XML file containing the worksheet data */
protected $dataXmlFilePath;
/**
* @param int $worksheetNumber Worksheet number, based on the order of appareance in [Content_Types].xml (zero-based)
/**\
* @param \Box\Spout\Reader\Sheet $externalSheet The associated "external" sheet
* @param int $worksheetIndex Worksheet index, based on the order of appareance in [Content_Types].xml (zero-based)
* @param string $dataXmlFilePath Path of the XML file containing the worksheet data
*/
public function __construct($worksheetNumber, $dataXmlFilePath)
public function __construct($externalSheet, $worksheetIndex, $dataXmlFilePath)
{
$this->worksheetNumber = $worksheetNumber;
$this->externalSheet = $externalSheet;
$this->worksheetIndex = $worksheetIndex;
$this->dataXmlFilePath = $dataXmlFilePath;
}
/**
* @return string Path of the XML file containing the worksheet data, without the leading slash
* @return string Path of the XML file containing the worksheet data,
* without the leading slash.
*/
public function getDataXmlFilePath()
{
return ltrim($this->dataXmlFilePath, '/');
}
/**
* @return \Box\Spout\Reader\Sheet The "external" sheet
*/
public function getExternalSheet()
{
return $this->externalSheet;
}
/**
* @return int
*/
public function getWorksheetNumber()
public function getWorksheetIndex()
{
return $this->worksheetNumber;
return $this->worksheetIndex;
}
}
+57
View File
@@ -0,0 +1,57 @@
<?php
namespace Box\Spout\Reader;
/**
* Class Sheet
* Represents a worksheet within a XLSX file
*
* @package Box\Spout\Reader
*/
class Sheet
{
/** @var int ID of the sheet */
protected $id;
/** @var int Index of the sheet, based on order of creation (zero-based) */
protected $index;
/** @var string Name of the sheet */
protected $name;
/**
* @param int $sheetId ID of the sheet
* @param int $sheetIndex Index of the sheet, based on order of creation (zero-based)
* @param string $sheetName Name of the sheet
*/
function __construct($sheetId, $sheetIndex, $sheetName)
{
$this->id = $sheetId;
$this->index = $sheetIndex;
$this->name = $sheetName;
}
/**
* @return int ID of the sheet
*/
public function getId()
{
return $this->id;
}
/**
* @return int Index of the sheet, based on order of creation (zero-based)
*/
public function getIndex()
{
return $this->index;
}
/**
* @return string Name of the sheet
*/
public function getName()
{
return $this->name;
}
}
+182 -41
View File
@@ -19,6 +19,14 @@ use Box\Spout\Reader\Helper\XLSX\WorksheetHelper;
*/
class XLSX extends AbstractReader
{
const CELL_TYPE_INLINE_STRING = 'inlineStr';
const CELL_TYPE_STR = 'str';
const CELL_TYPE_SHARED_STRING = 's';
const CELL_TYPE_BOOLEAN = 'b';
const CELL_TYPE_NUMERIC = 'n';
const CELL_TYPE_DATE = 'd';
const CELL_TYPE_ERROR = 'e';
/** @var string Real path of the file to read */
protected $filePath;
@@ -72,11 +80,15 @@ class XLSX extends AbstractReader
$this->zip = new \ZipArchive();
if ($this->zip->open($filePath) === true) {
// Extracts all the strings from the worksheets for easy access in the future
$this->extractSharedStrings($filePath);
$this->sharedStringsHelper = new SharedStringsHelper($filePath, $this->tempFolder);
if ($this->sharedStringsHelper->hasSharedStrings()) {
// Extracts all the strings from the worksheets for easy access in the future
$this->sharedStringsHelper->extractSharedStrings();
}
// Fetch all available worksheets
$this->worksheetHelper = new WorksheetHelper($filePath);
$this->worksheetHelper = new WorksheetHelper($filePath, $this->globalFunctionsHelper);
$this->worksheets = $this->worksheetHelper->getWorksheets($filePath);
if (count($this->worksheets) === 0) {
@@ -87,19 +99,6 @@ class XLSX extends AbstractReader
}
}
/**
* Builds an in-memory array containing all the shared strings of the worksheets.
*
* @param string $filePath Path of the XLSX file to be read
* @return void
* @throws \Box\Spout\Common\Exception\IOException If sharedStrings XML file can't be read
*/
protected function extractSharedStrings($filePath)
{
$this->sharedStringsHelper = new SharedStringsHelper($filePath, $this->tempFolder);
$this->sharedStringsHelper->extractSharedStrings();
}
/**
* Returns whether another worksheet exists after the current worksheet.
*
@@ -119,29 +118,31 @@ class XLSX extends AbstractReader
* Moves the pointer to the current worksheet.
* Moving to another worksheet will stop the reading in the current worksheet.
*
* @return void
* @return \Box\Spout\Reader\Sheet The next sheet
* @throws Exception\ReaderNotOpenedException If the stream was not opened first
* @throws Exception\EndOfWorksheetsReachedException If there is no more worksheets to read
*/
public function nextSheet()
{
if ($this->hasNextSheet()) {
if ($this->currentWorksheet === null) {
$nextWorksheet = $this->worksheets[0];
} else {
$currentWorksheetNumber = $this->currentWorksheet->getWorksheetNumber();
$nextWorksheet = $this->worksheets[$currentWorksheetNumber + 1];
}
$this->initXmlReaderForWorksheetData($nextWorksheet);
$this->currentWorksheet = $nextWorksheet;
// make sure that we are ready to read more rows
$this->hasReachedEndOfFile = false;
$this->emptyRowDataBuffer();
} else {
if (!$this->hasNextSheet()) {
throw new EndOfWorksheetsReachedException('End of worksheets was reached. Cannot read more worksheets.');
}
if ($this->currentWorksheet === null) {
$nextWorksheet = $this->worksheets[0];
} else {
$currentWorksheetIndex = $this->currentWorksheet->getWorksheetIndex();
$nextWorksheet = $this->worksheets[$currentWorksheetIndex + 1];
}
$this->initXmlReaderForWorksheetData($nextWorksheet);
$this->currentWorksheet = $nextWorksheet;
// make sure that we are ready to read more rows
$this->hasReachedEndOfFile = false;
$this->emptyRowDataBuffer();
return $this->currentWorksheet->getExternalSheet();
}
/**
@@ -184,6 +185,7 @@ class XLSX extends AbstractReader
throw new BadUsageException('You must call nextSheet() before calling hasNextRow() or nextRow()');
}
$escaper = new \Box\Spout\Common\Escaper\XLSX();
$isInsideRowTag = false;
$rowData = [];
@@ -195,6 +197,7 @@ class XLSX extends AbstractReader
$lastCellIndex = $matches[1];
$this->numberOfColumns = CellHelper::getColumnIndexFromCellIndex($lastCellIndex) + 1;
}
} else if ($this->xmlReader->nodeType == \XMLReader::ELEMENT && $this->xmlReader->name === 'row') {
// Start of the row description
$isInsideRowTag = true;
@@ -207,20 +210,15 @@ class XLSX extends AbstractReader
$numberOfColumnsForRow = intval($numberOfColumnsForRow);
}
$rowData = ($numberOfColumnsForRow !== 0) ? array_fill(0, $numberOfColumnsForRow, '') : [];
} else if ($isInsideRowTag && $this->xmlReader->nodeType == \XMLReader::ELEMENT && $this->xmlReader->name === 'c') {
// Start of a cell description
$currentCellIndex = $this->xmlReader->getAttribute('r');
$currentColumnIndex = CellHelper::getColumnIndexFromCellIndex($currentCellIndex);
$node = $this->xmlReader->expand();
$hasSharedString = ($this->xmlReader->getAttribute('t') === 's');
if ($hasSharedString) {
$sharedStringIndex = intval($node->nodeValue);
$rowData[$currentColumnIndex] = $this->sharedStringsHelper->getStringAtIndex($sharedStringIndex);
} else {
// for inline strings or numbers, just get the value
$rowData[$currentColumnIndex] = trim($node->nodeValue);
}
$node = $this->xmlReader->expand();
$rowData[$currentColumnIndex] = $this->getCellValue($node, $escaper);
} else if ($this->xmlReader->nodeType == \XMLReader::END_ELEMENT && $this->xmlReader->name === 'row') {
// End of the row description
// If needed, we fill the empty cells
@@ -233,6 +231,149 @@ class XLSX extends AbstractReader
return ($rowData !== []) ? $rowData : null;
}
/**
* Returns the cell's string value from a node's nested value node
*
* @param \DOMNode $node
* @return string The value associated with the cell
*/
protected function getVNodeValue($node)
{
// for cell types having a "v" tag containing the value.
// if not, the returned value should be empty string.
$vNode = $node->getElementsByTagName('v')->item(0);
if ($vNode !== null) {
return $vNode->nodeValue;
}
return "";
}
/**
* Returns the cell String value where string is inline.
*
* @param \DOMNode $node
* @param \Box\Spout\Common\Escaper\XLSX $escaper
* @return string The value associated with the cell (null when the cell has an error)
*/
protected function formatInlineStringCellValue($node, $escaper)
{
// inline strings are formatted this way:
// <c r="A1" t="inlineStr"><is><t>[INLINE_STRING]</t></is></c>
$tNode = $node->getElementsByTagName('t')->item(0);
$escapedCellValue = trim($tNode->nodeValue);
$cellValue = $escaper->unescape($escapedCellValue);
return $cellValue;
}
/**
* Returns the cell String value from shared-strings file using nodeValue index.
*
* @param string $nodeValue
* @param \Box\Spout\Common\Escaper\XLSX $escaper
* @return string The value associated with the cell (null when the cell has an error)
*/
protected function formatSharedStringCellValue($nodeValue, $escaper)
{
// shared strings are formatted this way:
// <c r="A1" t="s"><v>[SHARED_STRING_INDEX]</v></c>
$sharedStringIndex = intval($nodeValue);
$escapedCellValue = $this->sharedStringsHelper->getStringAtIndex($sharedStringIndex);
$cellValue = $escaper->unescape($escapedCellValue);
return $cellValue;
}
/**
* Returns the cell String value, where string is stored in value node.
*
* @param string $nodeValue
* @param \Box\Spout\Common\Escaper\XLSX $escaper
* @return string The value associated with the cell (null when the cell has an error)
*/
protected function formatStrCellValue($nodeValue, $escaper)
{
$escapedCellValue = trim($nodeValue);
$cellValue = $escaper->unescape($escapedCellValue);
return $cellValue;
}
/**
* Returns the cell Numeric value from string of nodeValue.
*
* @param string $nodeValue
* @param \Box\Spout\Common\Escaper\XLSX $escaper
* @return int|float The value associated with the cell
*/
protected function formatNumericCellValue($nodeValue)
{
$cellValue = is_int($nodeValue) ? intval($nodeValue) : floatval($nodeValue);
return $cellValue;
}
/**
* Returns the cell Boolean value from a specific node's Value.
*
* @param string $nodeValue
* @return bool The value associated with the cell
*/
protected function formatBooleanCellValue($nodeValue)
{
// !! is similar to boolval()
$cellValue = !!$nodeValue;
return $cellValue;
}
/**
* Returns a cell's PHP Date value, associated to the given stored nodeValue.
*
* @param string $nodeValue
* @param \Box\Spout\Common\Escaper\XLSX $escaper
* @return DateTime|null The value associated with the cell (null when the cell has an error)
*/
protected function formatDateCellValue($nodeValue)
{
try { // Mitigate thrown Exception on invalid date-time format (http://php.net/manual/en/datetime.construct.php)
$cellValue = new \DateTime($nodeValue);
return $cellValue;
} catch (\Exception $e) {
return null;
}
}
/**
* Returns the (unescaped) correctly marshalled, cell value associated to the given XML node.
*
* @param \DOMNode $node
* @param \Box\Spout\Common\Escaper\XLSX $escaper
* @return string|int|float|bool|null The value associated with the cell (null when the cell has an error)
*/
protected function getCellValue($node, $escaper)
{
// Default cell type is "n"
$cellType = $node->getAttribute('t') ?: 'n';
$vNodeValue = $this->getVNodeValue($node);
if ( ($vNodeValue === "") && ($cellType !== self::CELL_TYPE_INLINE_STRING) ) {
return $vNodeValue;
}
switch ($cellType)
{
case self::CELL_TYPE_INLINE_STRING:
return $this->formatInlineStringCellValue($node, $escaper);
case self::CELL_TYPE_SHARED_STRING:
return $this->formatSharedStringCellValue($vNodeValue, $escaper);
case self::CELL_TYPE_STR:
return $this->formatStrCellValue($vNodeValue, $escaper);
case self::CELL_TYPE_BOOLEAN:
return $this->formatBooleanCellValue($vNodeValue);
case self::CELL_TYPE_NUMERIC:
return $this->formatNumericCellValue($vNodeValue);
case self::CELL_TYPE_DATE:
return $this->formatDateCellValue($vNodeValue);
default:
return null;
}
}
/**
* Closes the reader. To be used after reading the file.
*
+2
View File
@@ -89,6 +89,8 @@ abstract class AbstractWriter implements WriterInterface
* Inits the writer and opens it to accept data.
* By using this method, the data will be outputted directly to the browser.
*
* @codeCoverageIgnore
*
* @param string $outputFileName Name of the output file that will contain the data. If a path is passed in, only the file name will be kept
* @return \Box\Spout\Writer\AbstractWriter
* @throws \Box\Spout\Common\Exception\IOException If the writer cannot be opened
+1 -1
View File
@@ -73,7 +73,7 @@ class CSV extends AbstractWriter
*/
protected function addRowToWriter(array $dataRow)
{
$wasWriteSuccessful = fputcsv($this->filePointer, $dataRow, $this->fieldDelimiter, $this->fieldEnclosure);
$wasWriteSuccessful = $this->globalFunctionsHelper->fputcsv($this->filePointer, $dataRow, $this->fieldDelimiter, $this->fieldEnclosure);
if ($wasWriteSuccessful === false) {
throw new IOException('Unable to write data');
}
@@ -36,7 +36,7 @@ EOD;
*/
public function __construct($xlFolder)
{
$sharedStringsFilePath = $xlFolder . DIRECTORY_SEPARATOR . self::SHARED_STRINGS_FILE_NAME;
$sharedStringsFilePath = $xlFolder . '/' . self::SHARED_STRINGS_FILE_NAME;
$this->sharedStringsFilePointer = fopen($sharedStringsFilePath, 'w');
$this->throwIfSharedStringsFilePointerIsNotAvailable();
+15 -3
View File
@@ -33,18 +33,30 @@ class ZipHelper
*/
protected function addFolderToZip($zip, $folderPath)
{
$folderRealPath = realpath($folderPath) . DIRECTORY_SEPARATOR;
$folderRealPath = $this->getNormalizedRealPath($folderPath) . '/';
$itemIterator = new \RecursiveIteratorIterator(new \RecursiveDirectoryIterator($folderPath, \RecursiveDirectoryIterator::SKIP_DOTS), \RecursiveIteratorIterator::SELF_FIRST);
foreach ($itemIterator as $itemInfo) {
$itemRealPath = realpath($itemInfo->getPathname());
$itemRealPath = $this->getNormalizedRealPath($itemInfo->getPathname());
$itemLocalPath = str_replace($folderRealPath, '', $itemRealPath);
if ($itemInfo->isFile()) {
$zip->addFile($itemInfo->getPathname(), $itemLocalPath);
$zip->addFile($itemRealPath, $itemLocalPath);
} else if ($itemInfo->isDir()) {
$zip->addEmptyDir($itemLocalPath);
}
}
}
/**
* Returns canonicalized absolute pathname, containing only forward slashes.
*
* @param string $path Path to normalize
* @return string Normalized and canonicalized path
*/
protected function getNormalizedRealPath($path)
{
$realPath = realpath($path);
return str_replace(DIRECTORY_SEPARATOR, '/', $realPath);
}
}
+2 -2
View File
@@ -67,8 +67,8 @@ class Workbook
*/
public function addNewSheet()
{
$newSheetNumber = count($this->worksheets);
$sheet = new Sheet($newSheetNumber);
$newSheetIndex = count($this->worksheets);
$sheet = new Sheet($newSheetIndex);
$worksheetFilesFolder = $this->fileSystemHelper->getXlWorksheetsFolder();
$worksheet = new Worksheet($sheet, $worksheetFilesFolder, $this->sharedStringsHelper, $this->shouldUseInlineStrings);
+3 -3
View File
@@ -56,7 +56,7 @@ EOD;
$this->stringsEscaper = new \Box\Spout\Common\Escaper\XLSX();
$this->worksheetFilePath = $worksheetFilesFolder . DIRECTORY_SEPARATOR . strtolower($this->externalSheet->getName()) . '.xml';
$this->worksheetFilePath = $worksheetFilesFolder . '/' . strtolower($this->externalSheet->getName()) . '.xml';
$this->startSheet();
}
@@ -96,8 +96,8 @@ EOD;
*/
public function getId()
{
// sheet number is zero-based, while ID is 1-based
return $this->externalSheet->getSheetNumber() + 1;
// sheet index is zero-based, while ID is 1-based
return $this->externalSheet->getIndex() + 1;
}
/**
+19 -9
View File
@@ -12,27 +12,27 @@ class Sheet
{
const DEFAULT_SHEET_NAME_PREFIX = 'Sheet';
/** @var int Number of the sheet, based on order of creation (zero-based) */
protected $sheetNumber;
/** @var int Index of the sheet, based on order of creation (zero-based) */
protected $index;
/** @var string Name of the sheet */
protected $name;
/**
* @param $sheetNumber Number of the sheet, based on order of creation (zero-based)
* @param int $sheetIndex Index of the sheet, based on order of creation (zero-based)
*/
function __construct($sheetNumber)
function __construct($sheetIndex)
{
$this->sheetNumber = $sheetNumber;
$this->name = self::DEFAULT_SHEET_NAME_PREFIX . ($sheetNumber + 1);
$this->index = $sheetIndex;
$this->name = self::DEFAULT_SHEET_NAME_PREFIX . ($sheetIndex + 1);
}
/**
* @return int Number of the sheet, based on order of creation (zero-based)
* @return int Index of the sheet, based on order of creation (zero-based)
*/
public function getSheetNumber()
public function getIndex()
{
return $this->sheetNumber;
return $this->index;
}
/**
@@ -42,4 +42,14 @@ class Sheet
{
return $this->name;
}
/**
* @param string $name Name of the sheet
* @return \Box\Spout\Writer\Sheet
*/
public function setName($name)
{
$this->name = $name;
return $this;
}
}
+52 -2
View File
@@ -19,11 +19,61 @@ class CSVTest extends \PHPUnit_Framework_TestCase
*
* @return void
*/
public function testReadShouldThrowExceptionIfFileDoesNotExist()
public function testOpenShouldThrowExceptionIfFileDoesNotExist()
{
$this->getAllRowsForFile('/path/to/fake/file.csv');
ReaderFactory::create(Type::CSV)->open('/path/to/fake/file.csv');
}
/**
* @expectedException \Box\Spout\Common\Exception\IOException
*
* @return void
*/
public function testOpenShouldThrowExceptionIfFileNotReadable()
{
$helperStub = $this->getMockBuilder('\Box\Spout\Common\Helper\GlobalFunctionsHelper')
->setMethods(['is_readable'])
->getMock();
$helperStub->method('is_readable')->willReturn(false);
$resourcePath = $this->getResourcePath('csv_standard.csv');
$reader = ReaderFactory::create(Type::CSV);
$reader->setGlobalFunctionsHelper($helperStub);
$reader->open($resourcePath);
}
/**
* @expectedException \Box\Spout\Reader\Exception\ReaderNotOpenedException
*
* @return void
*/
public function testReadShouldThrowExceptionIfReadBeforeReaderOpened()
{
$reader = ReaderFactory::create(Type::CSV);
$reader->hasNextRow();
}
/**
* @expectedException \Box\Spout\Reader\Exception\EndOfFileReachedException
*
* @return void
*/
public function testReadShouldThrowExceptionIfNextRowCalledAfterReadingDone()
{
$resourcePath = $this->getResourcePath('csv_standard.csv');
$reader = ReaderFactory::create(Type::CSV);
$reader->open($resourcePath);
while ($reader->hasNextRow()) {
$reader->nextRow();
}
$reader->nextRow();
}
/**
* @return void
*/
@@ -0,0 +1,60 @@
<?php
namespace Box\Spout\Reader\Helper\XLSX;
/**
* Class CellHelperTest
*
* @package Box\Spout\Reader\Helper\XLSX
*/
class CellHelperTest extends \PHPUnit_Framework_TestCase
{
/**
* @return void
*/
public function testFillMissingArrayIndexes()
{
$arrayToFill = [1 => 1, 3 => 3];
$filledArray = CellHelper::fillMissingArrayIndexes($arrayToFill, 'FILL');
$expectedFilledArray = ['FILL', 1, 'FILL', 3];
$this->assertEquals($expectedFilledArray, $filledArray);
}
/**
* @return array
*/
public function dataProviderForTestGetColumnIndexFromCellIndex()
{
return [
['A1', 0],
['Z3', 25],
['AA5', 26],
['AB24', 27],
['BC5', 54],
['BCZ99', 1455],
];
}
/**
* @dataProvider dataProviderForTestGetColumnIndexFromCellIndex
*
* @param string $cellIndex
* @param int $expectedColumnIndex
* @return void
*/
public function testGetColumnIndexFromCellIndex($cellIndex, $expectedColumnIndex)
{
$this->assertEquals($expectedColumnIndex, CellHelper::getColumnIndexFromCellIndex($cellIndex));
}
/**
* @expectedException \Box\Spout\Common\Exception\InvalidArgumentException
*
* @return void
*/
public function testGetColumnIndexFromCellIndexShouldThrowIfInvalidCellIndex()
{
CellHelper::getColumnIndexFromCellIndex('InvalidCellIndex');
}
}
@@ -0,0 +1,99 @@
<?php
namespace Box\Spout\Reader\Helper\XLSX\SharedStringsCaching;
/**
* Class CachingStrategyFactoryTest
*
* @package Box\Spout\Reader\Helper\XLSX\SharedStringsCaching
*/
class CachingStrategyFactoryTest extends \PHPUnit_Framework_TestCase
{
/**
* @return array
*/
public function dataProviderForTestGetBestCachingStrategy()
{
return [
[CachingStrategyFactory::MAX_NUM_STRINGS_PER_TEMP_FILE, -1, 'FileBasedStrategy'],
[CachingStrategyFactory::MAX_NUM_STRINGS_PER_TEMP_FILE + 10, -1, 'FileBasedStrategy'],
[CachingStrategyFactory::MAX_NUM_STRINGS_PER_TEMP_FILE - 10, -1, 'InMemoryStrategy'],
[10 , CachingStrategyFactory::AMOUNT_MEMORY_NEEDED_PER_STRING_IN_KB * 10, 'FileBasedStrategy'],
[15, CachingStrategyFactory::AMOUNT_MEMORY_NEEDED_PER_STRING_IN_KB * 10, 'FileBasedStrategy'],
[5 , CachingStrategyFactory::AMOUNT_MEMORY_NEEDED_PER_STRING_IN_KB * 10, 'InMemoryStrategy'],
];
}
/**
* @dataProvider dataProviderForTestGetBestCachingStrategy
*
* @param int $sharedStringsUniqueCount
* @param int $memoryLimitInKB
* @param string $expectedStrategyClassName
* @return void
*/
public function testGetBestCachingStrategy($sharedStringsUniqueCount, $memoryLimitInKB, $expectedStrategyClassName)
{
/** @var CachingStrategyFactory|\PHPUnit_Framework_MockObject_MockObject $factoryStub */
$factoryStub = $this
->getMockBuilder('\Box\Spout\Reader\Helper\XLSX\SharedStringsCaching\CachingStrategyFactory')
->disableOriginalConstructor()
->setMethods(['getMemoryLimitInKB'])
->getMock();
$factoryStub->method('getMemoryLimitInKB')->willReturn($memoryLimitInKB);
\ReflectionHelper::setStaticValue('\Box\Spout\Reader\Helper\XLSX\SharedStringsCaching\CachingStrategyFactory', 'instance', $factoryStub);
$strategy = $factoryStub->getBestCachingStrategy($sharedStringsUniqueCount, null);
$fullExpectedStrategyClassName = 'Box\Spout\Reader\Helper\XLSX\SharedStringsCaching\\' . $expectedStrategyClassName;
$this->assertEquals($fullExpectedStrategyClassName, get_class($strategy));
$strategy->clearCache();
\ReflectionHelper::reset();
}
/**
* @return array
*/
public function dataProviderForTestGetMemoryLimitInKB()
{
return [
['-1', -1],
['invalid', -1],
['1024B', 1],
['128K', 128],
['256KB', 256],
['512M', 512 * 1024],
['2MB', 2 * 1024],
['1G', 1 * 1024 * 1024],
['10GB', 10 * 1024 * 1024],
['2T', 2 * 1024 * 1024 * 1024],
['5TB', 5 * 1024 * 1024 * 1024],
];
}
/**
* @dataProvider dataProviderForTestGetMemoryLimitInKB
*
* @param string $memoryLimitFormatted
* @param float $expectedMemoryLimitInKB
* @return void
*/
public function testGetMemoryLimitInKB($memoryLimitFormatted, $expectedMemoryLimitInKB)
{
/** @var CachingStrategyFactory|\PHPUnit_Framework_MockObject_MockObject $factoryStub */
$factoryStub = $this
->getMockBuilder('\Box\Spout\Reader\Helper\XLSX\SharedStringsCaching\CachingStrategyFactory')
->disableOriginalConstructor()
->setMethods(['getMemoryLimitFromIni'])
->getMock();
$factoryStub->method('getMemoryLimitFromIni')->willReturn($memoryLimitFormatted);
$memoryLimitInKB = \ReflectionHelper::callMethodOnObject($factoryStub, 'getMemoryLimitInKB');
$this->assertEquals($expectedMemoryLimitInKB, $memoryLimitInKB);
}
}
@@ -2,6 +2,9 @@
namespace Box\Spout\Reader\Helper\XLSX;
use Box\Spout\Reader\Helper\XLSX\SharedStringsCaching\CachingStrategyFactory;
use Box\Spout\Reader\Helper\XLSX\SharedStringsCaching\FileBasedStrategy;
use Box\Spout\Reader\Helper\XLSX\SharedStringsCaching\InMemoryStrategy;
use Box\Spout\TestUsingResource;
/**
@@ -33,46 +36,6 @@ class SharedStringsHelperTest extends \PHPUnit_Framework_TestCase
$this->sharedStringsHelper->cleanup();
}
/**
* @return void
*/
public function testExtractSharedStringsShouldCreateTempFileWithSharedStrings()
{
$this->sharedStringsHelper->extractSharedStrings();
$tempFolder = \ReflectionHelper::getValueOnObject($this->sharedStringsHelper, 'tempFolder');
$filesInTempFolder = $this->getFilesInFolder($tempFolder);
$this->assertEquals(1, count($filesInTempFolder), 'One temp file should have been created in the temp folder.');
$tempFileContents = file_get_contents($filesInTempFolder[0]);
$tempFileContentsPerLine = explode(PHP_EOL, $tempFileContents);
$this->assertEquals('s1--A1', $tempFileContentsPerLine[0]);
$this->assertEquals('s1--E5', $tempFileContentsPerLine[24]);
}
/**
* Returns all files that are in the given folder.
* It does not include "." and ".." and is not recursive.
*
* @param string $folderPath
* @return array
*/
private function getFilesInFolder($folderPath)
{
$files = [];
$directoryIterator = new \DirectoryIterator($folderPath);
foreach ($directoryIterator as $fileInfo) {
if ($fileInfo->isFile()) {
$files[] = $fileInfo->getPathname();
}
}
return $files;
}
/**
* @expectedException \Box\Spout\Reader\Exception\SharedStringNotFoundException
* @return void
@@ -95,6 +58,9 @@ class SharedStringsHelperTest extends \PHPUnit_Framework_TestCase
$sharedString = $this->sharedStringsHelper->getStringAtIndex(24);
$this->assertEquals('s1--E5', $sharedString);
$usedCachingStrategy = \ReflectionHelper::getValueOnObject($this->sharedStringsHelper, 'cachingStrategy');
$this->assertTrue($usedCachingStrategy instanceof InMemoryStrategy);
}
/**
@@ -115,4 +81,32 @@ class SharedStringsHelperTest extends \PHPUnit_Framework_TestCase
$sharedStringsHelper->cleanup();
}
/**
* @return void
*/
public function testGetStringAtIndexWithFileBasedStrategy()
{
// force the file-based strategy by setting no memory limit
$originalMemoryLimit = ini_get('memory_limit');
ini_set('memory_limit', '-1');
$resourcePath = $this->getResourcePath('sheet_with_lots_of_shared_strings.xlsx');
$sharedStringsHelper = new SharedStringsHelper($resourcePath);
$sharedStringsHelper->extractSharedStrings();
$sharedString = $sharedStringsHelper->getStringAtIndex(0);
$this->assertEquals('str', $sharedString);
$sharedString = $sharedStringsHelper->getStringAtIndex(CachingStrategyFactory::MAX_NUM_STRINGS_PER_TEMP_FILE + 1);
$this->assertEquals('str', $sharedString);
$usedCachingStrategy = \ReflectionHelper::getValueOnObject($sharedStringsHelper, 'cachingStrategy');
$this->assertTrue($usedCachingStrategy instanceof FileBasedStrategy);
$sharedStringsHelper->cleanup();
ini_set('memory_limit', $originalMemoryLimit);
}
}
+21
View File
@@ -0,0 +1,21 @@
<?php
namespace Box\Spout\Reader;
/**
* Class ReaderFactoryTest
*
* @package Box\Spout\Writer
*/
class ReaderFactoryTest extends \PHPUnit_Framework_TestCase
{
/**
* @expectedException \Box\Spout\Common\Exception\UnsupportedTypeException
*
* @return void
*/
public function testCreateReaderShouldThrowWithUnsupportedType()
{
ReaderFactory::create('unsupportedType');
}
}
+52
View File
@@ -0,0 +1,52 @@
<?php
namespace Box\Spout\Reader;
use Box\Spout\Common\Type;
use Box\Spout\TestUsingResource;
/**
* Class SheetTest
*
* @package Box\Spout\Reader
*/
class SheetTest extends \PHPUnit_Framework_TestCase
{
use TestUsingResource;
/**
* @return void
*/
public function testNextSheetShouldReturnCorrectSheetInfos()
{
$sheets = $this->openFileAndReturnSheets('two_sheets_with_custom_names.xlsx');
$this->assertEquals('CustomName1', $sheets[0]->getName());
$this->assertEquals(0, $sheets[0]->getIndex());
$this->assertEquals(1, $sheets[0]->getId());
$this->assertEquals('CustomName2', $sheets[1]->getName());
$this->assertEquals(1, $sheets[1]->getIndex());
$this->assertEquals(2, $sheets[1]->getId());
}
/**
* @param string $fileName
* @return Sheet[]
*/
private function openFileAndReturnSheets($fileName)
{
$resourcePath = $this->getResourcePath($fileName);
$reader = ReaderFactory::create(Type::XLSX);
$reader->open($resourcePath);
$sheets = [];
while ($reader->hasNextSheet()) {
$sheets[] = $reader->nextSheet();
}
$reader->close();
return $sheets;
}
}
+137 -6
View File
@@ -2,6 +2,7 @@
namespace Box\Spout\Reader;
use Box\Spout\Common\Exception\IOException;
use Box\Spout\Common\Type;
use Box\Spout\TestUsingResource;
@@ -38,6 +39,37 @@ class XLSXTest extends \PHPUnit_Framework_TestCase
$this->getAllRowsForFile($filePath);
}
/**
* @expectedException \Box\Spout\Reader\Exception\ReaderNotOpenedException
*
* @return void
*/
public function testHasNextSheetShouldThrowExceptionIfReaderNotOpened()
{
$reader = ReaderFactory::create(Type::XLSX);
$reader->hasNextSheet();
}
/**
* @expectedException \Box\Spout\Reader\Exception\EndOfWorksheetsReachedException
*
* @return void
*/
public function testNextSheetShouldThrowExceptionIfNoMoreSheetsToRead()
{
$fileName = 'one_sheet_with_shared_strings.xlsx';
$resourcePath = $this->getResourcePath($fileName);
$reader = ReaderFactory::create(Type::XLSX);
$reader->open($resourcePath);
while ($reader->hasNextSheet()) {
$reader->nextSheet();
}
$reader->nextSheet();
}
/**
* @return array
*/
@@ -69,6 +101,43 @@ class XLSXTest extends \PHPUnit_Framework_TestCase
}
}
/**
* @return void
*/
public function testReadShouldSupportFilesWithoutSharedStringsFile()
{
$allRows = $this->getAllRowsForFile('sheet_with_no_shared_strings_file.xlsx');
$expectedRows = [
[10, 11],
[20, 21],
];
$this->assertEquals($expectedRows, $allRows);
}
/**
* @return void
*/
public function testReadShouldSupportAllCellTypes()
{
$allRows = $this->getAllRowsForFile('sheet_with_all_cell_types.xlsx');
$expectedRows = [
[
's1--A1', 's1--A2',
false, true,
\DateTime::createFromFormat('Y-m-d H:i:s', '2015-06-03 13:21:58'),
\DateTime::createFromFormat('Y-m-d H:i:s', '2015-06-01 00:00:00'),
10, 10.43,
null,
'weird string', // valid 'str' string
null, // invalid date
],
['', '', '', '', '', '', '', '', ''],
];
$this->assertEquals($expectedRows, $allRows);
}
/**
* @return void
*/
@@ -140,6 +209,32 @@ class XLSXTest extends \PHPUnit_Framework_TestCase
$this->assertEquals($expectedRows, $allRows);
}
/**
* @return void
*/
public function testReadShouldSupportEmptySharedString()
{
$allRows = $this->getAllRowsForFile('sheet_with_empty_shared_string.xlsx');
$expectedRows = [
['s1--A1', '', 's1--C1'],
];
$this->assertEquals($expectedRows, $allRows);
}
/**
* @return void
*/
public function testReadShouldPreserveSpaceIfSpecified()
{
$allRows = $this->getAllRowsForFile('sheet_with_preserve_space_shared_strings.xlsx');
$expectedRows = [
[' s1--A1', 's1--B1 ', ' s1--C1 '],
];
$this->assertEquals($expectedRows, $allRows);
}
/**
* @return void
*/
@@ -151,18 +246,39 @@ class XLSXTest extends \PHPUnit_Framework_TestCase
$this->assertEquals($expectedRow, $allRows[0], 'Pronunciation data should be removed.');
}
/**
* @return array
*/
public function dataProviderForTestReadShouldBeProtectedAgainstAttacks()
{
return [
['attack_billion_laughs.xlsx'],
['attack_quadratic_blowup.xlsx'],
];
}
/**
* @dataProvider dataProviderForTestReadShouldBeProtectedAgainstAttacks
* @NOTE: The LIBXML_NOENT is used to ACTUALLY substitute entities (and should therefore not be used)
*
* @param string $fileName
* @return void
*/
public function testReadShouldBeProtectedAgainstBillionLaughsAttack()
public function testReadShouldBeProtectedAgainstAttacks($fileName)
{
$allRows = $this->getAllRowsForFile('billion_laughs_test_file.xlsx');
$startTime = microtime(true);
$expectedMaxMemoryUsage = 10 * 1024 * 1024; // 10MB
$this->assertLessThan($expectedMaxMemoryUsage, memory_get_peak_usage(true), 'Entities should not be expanded and therefore consume all the memory.');
try {
$this->getAllRowsForFile($fileName);
$this->fail('An exception should have been thrown');
} catch (IOException $exception) {
$duration = microtime(true) - $startTime;
$this->assertLessThan(10, $duration, 'Entities should not be expanded and therefore take more than 10 seconds to be parsed.');
$expectedFirstRow = ['s1--A1', 's1--B1', 's1--C1', 's1--D1', 's1--E1'];
$this->assertEquals($expectedFirstRow, $allRows[0], 'Entities should be ignored when reading XML files.');
$expectedMaxMemoryUsage = 30 * 1024 * 1024; // 30MB
$this->assertLessThan($expectedMaxMemoryUsage, memory_get_peak_usage(true), 'Entities should not be expanded and therefore consume all the memory.');
}
}
/**
@@ -174,6 +290,21 @@ class XLSXTest extends \PHPUnit_Framework_TestCase
$this->assertEquals([], $allRows, 'Sheet with no cells should be correctly processed.');
}
/**
* @return void
*/
public function testReadShouldSkipFormulas()
{
$allRows = $this->getAllRowsForFile('sheet_with_formulas.xlsx');
$expectedRows = [
['val1', 'val2', 'total1', 'total2'],
[10, 20, 30, 21],
[11, 21, 32, 41],
];
$this->assertEquals($expectedRows, $allRows);
}
/**
* @param string $fileName
* @return array All the read rows the given file
+23
View File
@@ -89,4 +89,27 @@ class ReflectionHelper
return $value;
}
/**
* Invoke a the given public or protected method on the given object.
*
* @param object $object
* @param string $methodName
* @param *mixed|null $params
*
* @return mixed|null
*/
public static function callMethodOnObject($object, $methodName)
{
$params = func_get_args();
array_shift($params); // object
array_shift($params); // methodName
$className = get_class($object);
$class = new ReflectionClass($className);
$method = $class->getMethod($methodName);
$method->setAccessible(true);
return $method->invokeArgs($object, $params);
}
}
+10
View File
@@ -49,6 +49,16 @@ class CSVTest extends \PHPUnit_Framework_TestCase
$writer->close();
}
/**
* @expectedException \Box\Spout\Common\Exception\InvalidArgumentException
*/
public function testAddRowsShouldThrowExceptionIfRowsAreNotArrayOfArrays()
{
$writer = WriterFactory::create(Type::CSV);
$writer->addRows(['csv--11', 'csv--12']);
$writer->close();
}
/**
* @return void
*/
+40 -4
View File
@@ -17,13 +17,13 @@ class SheetTest extends \PHPUnit_Framework_TestCase
/**
* @return void
*/
public function testGetSheetNumber()
public function testGetSheetIndex()
{
$sheets = $this->writeDataAndReturnSheets('test_get_sheet_number.xlsx');
$sheets = $this->writeDataAndReturnSheets('test_get_sheet_index.xlsx');
$this->assertEquals(2, count($sheets), '2 sheets should have been created');
$this->assertEquals(0, $sheets[0]->getSheetNumber(), 'The first sheet should be number 0');
$this->assertEquals(1, $sheets[1]->getSheetNumber(), 'The second sheet should be number 1');
$this->assertEquals(0, $sheets[0]->getIndex(), 'The first sheet should be index 0');
$this->assertEquals(1, $sheets[1]->getIndex(), 'The second sheet should be index 1');
}
/**
@@ -38,6 +38,28 @@ class SheetTest extends \PHPUnit_Framework_TestCase
$this->assertEquals('Sheet2', $sheets[1]->getName(), 'Invalid name for the second sheet');
}
/**
* @return void
*/
public function testSetSheetNameShouldCreateSheetWithCustomName()
{
$fileName = 'test_set_name_should_create_sheet_with_custom_name.xlsx';
$this->createGeneratedFolderIfNeeded($fileName);
$resourcePath = $this->getGeneratedResourcePath($fileName);
$writer = WriterFactory::create(Type::XLSX);
$writer->openToFile($resourcePath);
$customSheetName = 'CustomName';
$sheet = $writer->getCurrentSheet();
$sheet->setName($customSheetName);
$writer->addRow(['xlsx--11', 'xlsx--12']);
$writer->close();
$this->assertSheetNameEquals($customSheetName, $resourcePath, "The sheet name should have been changed to '$customSheetName'");
}
/**
* @param string $fileName
* @return Sheet[]
@@ -59,4 +81,18 @@ class SheetTest extends \PHPUnit_Framework_TestCase
return $writer->getSheets();
}
/**
* @param string $expectedName
* @param string $resourcePath
* @param string $message
* @return void
*/
private function assertSheetNameEquals($expectedName, $resourcePath, $message = '')
{
$pathToWorkbookFile = $resourcePath . '#xl/workbook.xml';
$xmlContents = file_get_contents('zip://' . $pathToWorkbookFile);
$this->assertContains('<sheet name="' . $expectedName . '"', $xmlContents, $message);
}
}
+21
View File
@@ -0,0 +1,21 @@
<?php
namespace Box\Spout\Writer;
/**
* Class WriterFactoryTest
*
* @package Box\Spout\Writer
*/
class WriterFactoryTest extends \PHPUnit_Framework_TestCase
{
/**
* @expectedException \Box\Spout\Common\Exception\UnsupportedTypeException
*
* @return void
*/
public function testCreateWriterShouldThrowWithUnsupportedType()
{
WriterFactory::create('unsupportedType');
}
}
+6 -7
View File
@@ -343,7 +343,6 @@ class XLSXTest extends \PHPUnit_Framework_TestCase
$this->assertInlineDataWasWrittenToSheet($fileName, 1, 'control&#039;s _x0015_ &quot;character&quot;');
}
/**
* @param array $allRows
* @param string $fileName
@@ -401,15 +400,15 @@ class XLSXTest extends \PHPUnit_Framework_TestCase
/**
* @param string $fileName
* @param int $sheetNumber
* @param int $sheetIndex
* @param mixed $inlineData
* @param string $message
* @return void
*/
private function assertInlineDataWasWrittenToSheet($fileName, $sheetNumber, $inlineData, $message = '')
private function assertInlineDataWasWrittenToSheet($fileName, $sheetIndex, $inlineData, $message = '')
{
$resourcePath = $this->getGeneratedResourcePath($fileName);
$pathToSheetFile = $resourcePath . '#xl/worksheets/sheet' . $sheetNumber . '.xml';
$pathToSheetFile = $resourcePath . '#xl/worksheets/sheet' . $sheetIndex . '.xml';
$xmlContents = file_get_contents('zip://' . $pathToSheetFile);
$this->assertContains((string)$inlineData, $xmlContents, $message);
@@ -417,15 +416,15 @@ class XLSXTest extends \PHPUnit_Framework_TestCase
/**
* @param string $fileName
* @param int $sheetNumber
* @param int $sheetIndex
* @param mixed $inlineData
* @param string $message
* @return void
*/
private function assertInlineDataWasNotWrittenToSheet($fileName, $sheetNumber, $inlineData, $message = '')
private function assertInlineDataWasNotWrittenToSheet($fileName, $sheetIndex, $inlineData, $message = '')
{
$resourcePath = $this->getGeneratedResourcePath($fileName);
$pathToSheetFile = $resourcePath . '#xl/worksheets/sheet' . $sheetNumber . '.xml';
$pathToSheetFile = $resourcePath . '#xl/worksheets/sheet' . $sheetIndex . '.xml';
$xmlContents = file_get_contents('zip://' . $pathToSheetFile);
$this->assertNotContains((string)$inlineData, $xmlContents, $message);
Binary file not shown.
Binary file not shown.
Binary file not shown.