- 浏览: 17796 次
- 性别:
文章分类
最新评论
excel2007文件格式与之前版本不同,之前版本采用的是微软自己的存储格式。07版内容的存储采用XML格式,所以,理所当然的,对大数据量的 xlsx文件的读取采用的也是XML的处理方式SAX。
同之前的版本一样,大数据量文件的读取采用的是事件模型eventusermodel。usermodel模式需要将文件一次性全部读到内存中,07版的既然采用的存储模式是xml,解析用的DOM方式也是如此,这种模式操作简单,容易上手,但是对于大量数据占用的内存也是相当可观,在Eclipse中经常出现内存溢出。
下面就是采用eventusermodel对07excel文件读取。
同上篇,我将当前行的单元格数据存储到List中,抽象出 optRows 方法,该方法会在每行末尾时调用,方法参数为当前行索引curRow(int型)及存有行内单元格数据的List。继承类只需实现该行级方法即可。
经测试,对12万条数据,7M大小的文件也能正常运行。无需设置vm的内存空间。
excel读取采用的API为POI3.6,使用前先下载此包,若运行中出现其他依赖包不存在,请下载相应依赖包。
抽象类:XxlsAbstract ,作用:遍历excel文件,提供行级操作方法 optRows
package com.gaosheng.util.xls;
import java.io.InputStream;
import java.sql.SQLException;
import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
import org.apache.poi.xssf.eventusermodel.XSSFReader;
import org.apache.poi.xssf.model.SharedStringsTable;
import org.apache.poi.xssf.usermodel.XSSFRichTextString;
import org.apache.poi.openxml4j.opc.OPCPackage;
import org.xml.sax.Attributes;
import org.xml.sax.InputSource;
import org.xml.sax.SAXException;
import org.xml.sax.XMLReader;
import org.xml.sax.helpers.DefaultHandler;
import org.xml.sax.helpers.XMLReaderFactory;
/**
* XSSF and SAX (Event API)
*/
public abstract class XxlsAbstract extends DefaultHandler {
private SharedStringsTable sst;
private String lastContents;
private boolean nextIsString;
private int sheetIndex = -1;
private List<String> rowlist = new ArrayList<String>();
private int curRow = 0;
private int curCol = 0;
//excel记录行操作方法,以行索引和行元素列表为参数,对一行元素进行操作,元素为String类型
// public abstract void optRows(int curRow, List<String> rowlist) throws SQLException ;
//excel记录行操作方法,以sheet索引,行索引和行元素列表为参数,对sheet的一行元素进行操作,元素为String类型
public abstract void optRows(int sheetIndex,int curRow, List<String> rowlist) throws SQLException;
//只遍历一个sheet,其中sheetId为要遍历的sheet索引,从1开始,1-3
public void processOneSheet(String filename,int sheetId) throws Exception {
OPCPackage pkg = OPCPackage.open(filename);
XSSFReader r = new XSSFReader(pkg);
SharedStringsTable sst = r.getSharedStringsTable();
XMLReader parser = fetchSheetParser(sst);
// rId2 found by processing the Workbook
// 根据 rId# 或 rSheet# 查找sheet
InputStream sheet2 = r.getSheet("rId"+sheetId);
sheetIndex++;
InputSource sheetSource = new InputSource(sheet2);
parser.parse(sheetSource);
sheet2.close();
}
/**
* 遍历 excel 文件
*/
public void process(String filename) throws Exception {
OPCPackage pkg = OPCPackage.open(filename);
XSSFReader r = new XSSFReader(pkg);
SharedStringsTable sst = r.getSharedStringsTable();
XMLReader parser = fetchSheetParser(sst);
Iterator<InputStream> sheets = r.getSheetsData();
while (sheets.hasNext()) {
curRow = 0;
sheetIndex++;
InputStream sheet = sheets.next();
InputSource sheetSource = new InputSource(sheet);
parser.parse(sheetSource);
sheet.close();
}
}
public XMLReader fetchSheetParser(SharedStringsTable sst)
throws SAXException {
XMLReader parser = XMLReaderFactory
.createXMLReader("org.apache.xerces.parsers.SAXParser");
this.sst = sst;
parser.setContentHandler(this);
return parser;
}
public void startElement(String uri, String localName, String name,
Attributes attributes) throws SAXException {
// c => 单元格
if (name.equals("c")) {
// 如果下一个元素是 SST 的索引,则将nextIsString标记为true
String cellType = attributes.getValue("t");
if (cellType != null && cellType.equals("s")) {
nextIsString = true;
} else {
nextIsString = false;
}
}
// 置空
lastContents = "";
}
public void endElement(String uri, String localName, String name)
throws SAXException {
// 根据SST的索引值的到单元格的真正要存储的字符串
// 这时characters()方法可能会被调用多次
if (nextIsString) {
try {
int idx = Integer.parseInt(lastContents);
lastContents = new XSSFRichTextString(sst.getEntryAt(idx))
.toString();
} catch (Exception e) {
}
}
// v => 单元格的值,如果单元格是字符串则v标签的值为该字符串在SST中的索引
// 将单元格内容加入rowlist中,在这之前先去掉字符串前后的空白符
if (name.equals("v")) {
String value = lastContents.trim();
value = value.equals("")?" ":value;
rowlist.add(curCol, value);
curCol++;
}else {
//如果标签名称为 row ,这说明已到行尾,调用 optRows() 方法
if (name.equals("row")) {
try {
optRows(sheetIndex,curRow,rowlist);
} catch (SQLException e) {
e.printStackTrace();
}
rowlist.clear();
curRow++;
curCol = 0;
}
}
}
public void characters(char[] ch, int start, int length)
throws SAXException {
//得到单元格内容的值
lastContents += new String(ch, start, length);
}
}
package com.gaosheng.util.xls;
import java.io.InputStream;
import java.sql.SQLException;
import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
import org.apache.poi.xssf.eventusermodel.XSSFReader;
import org.apache.poi.xssf.model.SharedStringsTable;
import org.apache.poi.xssf.usermodel.XSSFRichTextString;
import org.apache.poi.openxml4j.opc.OPCPackage;
import org.xml.sax.Attributes;
import org.xml.sax.InputSource;
import org.xml.sax.SAXException;
import org.xml.sax.XMLReader;
import org.xml.sax.helpers.DefaultHandler;
import org.xml.sax.helpers.XMLReaderFactory;
/**
* XSSF and SAX (Event API)
*/
public abstract class XxlsAbstract extends DefaultHandler {
private SharedStringsTable sst;
private String lastContents;
private boolean nextIsString;
private int sheetIndex = -1;
private List<String> rowlist = new ArrayList<String>();
private int curRow = 0;
private int curCol = 0;
//excel记录行操作方法,以行索引和行元素列表为参数,对一行元素进行操作,元素为String类型
// public abstract void optRows(int curRow, List<String> rowlist) throws SQLException ;
//excel记录行操作方法,以sheet索引,行索引和行元素列表为参数,对sheet的一行元素进行操作,元素为String类型
public abstract void optRows(int sheetIndex,int curRow, List<String> rowlist) throws SQLException;
//只遍历一个sheet,其中sheetId为要遍历的sheet索引,从1开始,1-3
public void processOneSheet(String filename,int sheetId) throws Exception {
OPCPackage pkg = OPCPackage.open(filename);
XSSFReader r = new XSSFReader(pkg);
SharedStringsTable sst = r.getSharedStringsTable();
XMLReader parser = fetchSheetParser(sst);
// rId2 found by processing the Workbook
// 根据 rId# 或 rSheet# 查找sheet
InputStream sheet2 = r.getSheet("rId"+sheetId);
sheetIndex++;
InputSource sheetSource = new InputSource(sheet2);
parser.parse(sheetSource);
sheet2.close();
}
/**
* 遍历 excel 文件
*/
public void process(String filename) throws Exception {
OPCPackage pkg = OPCPackage.open(filename);
XSSFReader r = new XSSFReader(pkg);
SharedStringsTable sst = r.getSharedStringsTable();
XMLReader parser = fetchSheetParser(sst);
Iterator<InputStream> sheets = r.getSheetsData();
while (sheets.hasNext()) {
curRow = 0;
sheetIndex++;
InputStream sheet = sheets.next();
InputSource sheetSource = new InputSource(sheet);
parser.parse(sheetSource);
sheet.close();
}
}
public XMLReader fetchSheetParser(SharedStringsTable sst)
throws SAXException {
XMLReader parser = XMLReaderFactory
.createXMLReader("org.apache.xerces.parsers.SAXParser");
this.sst = sst;
parser.setContentHandler(this);
return parser;
}
public void startElement(String uri, String localName, String name,
Attributes attributes) throws SAXException {
// c => 单元格
if (name.equals("c")) {
// 如果下一个元素是 SST 的索引,则将nextIsString标记为true
String cellType = attributes.getValue("t");
if (cellType != null && cellType.equals("s")) {
nextIsString = true;
} else {
nextIsString = false;
}
}
// 置空
lastContents = "";
}
public void endElement(String uri, String localName, String name)
throws SAXException {
// 根据SST的索引值的到单元格的真正要存储的字符串
// 这时characters()方法可能会被调用多次
if (nextIsString) {
try {
int idx = Integer.parseInt(lastContents);
lastContents = new XSSFRichTextString(sst.getEntryAt(idx))
.toString();
} catch (Exception e) {
}
}
// v => 单元格的值,如果单元格是字符串则v标签的值为该字符串在SST中的索引
// 将单元格内容加入rowlist中,在这之前先去掉字符串前后的空白符
if (name.equals("v")) {
String value = lastContents.trim();
value = value.equals("")?" ":value;
rowlist.add(curCol, value);
curCol++;
}else {
//如果标签名称为 row ,这说明已到行尾,调用 optRows() 方法
if (name.equals("row")) {
try {
optRows(sheetIndex,curRow,rowlist);
} catch (SQLException e) {
e.printStackTrace();
}
rowlist.clear();
curRow++;
curCol = 0;
}
}
}
public void characters(char[] ch, int start, int length)
throws SAXException {
//得到单元格内容的值
lastContents += new String(ch, start, length);
}
}
继承类:XxlsBig,作用:将数据转出到数据库临时表
Java代码
package com.gaosheng.util.examples.xls;
import java.io.FileInputStream;
import java.io.IOException;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.List;
import java.util.Properties;
import com.gaosheng.util.xls.XxlsAbstract;
public class XxlsBig extends XxlsAbstract {
public static void main(String[] args) throws Exception {
XxlsBig howto = new XxlsBig("temp_table");
howto.processOneSheet("F:/new.xlsx",1);
howto.process("F:/new.xlsx");
howto.close();
}
public XxlsBig(String tableName) throws SQLException{
this.conn = getNew_Conn();
this.statement = conn.createStatement();
this.tableName = tableName;
}
private Connection conn = null;
private Statement statement = null;
private PreparedStatement newStatement = null;
private String tableName = "temp_table";
private boolean create = true;
public void optRows(int sheetIndex,int curRow, List<String> rowlist) throws SQLException {
if (sheetIndex == 0 && curRow == 0) {
StringBuffer preSql = new StringBuffer("insert into " + tableName
+ " values(");
StringBuffer table = new StringBuffer("create table " + tableName
+ "(");
int c = rowlist.size();
for (int i = 0; i < c; i++) {
preSql.append("?,");
table.append(rowlist.get(i));
table.append(" varchar2(100) ,");
}
table.deleteCharAt(table.length() - 1);
preSql.deleteCharAt(preSql.length() - 1);
table.append(")");
preSql.append(")");
if (create) {
statement = conn.createStatement();
try{
statement.execute("drop table "+tableName);
}catch(Exception e){
}finally{
System.out.println("表 "+tableName+" 删除成功");
}
if (!statement.execute(table.toString())) {
System.out.println("创建表 "+tableName+" 成功");
// return;
} else {
System.out.println("创建表 "+tableName+" 失败");
return;
}
}
conn.setAutoCommit(false);
newStatement = conn.prepareStatement(preSql.toString());
} else if(curRow>0) {
// 一般行
int col = rowlist.size();
for (int i = 0; i < col; i++) {
newStatement.setString(i + 1, rowlist.get(i).toString());
}
newStatement.addBatch();
if (curRow % 1000 == 0) {
newStatement.executeBatch();
conn.commit();
}
}
}
private static Connection getNew_Conn() {
Connection conn = null;
Properties props = new Properties();
FileInputStream fis = null;
try {
fis = new FileInputStream("D:/database.properties");
props.load(fis);
DriverManager.registerDriver(new oracle.jdbc.driver.OracleDriver());
// String jdbcURLString =
// "jdbc:oracle:thin:@192.168.0.28:1521:orcl";
StringBuffer jdbcURLString = new StringBuffer();
jdbcURLString.append("jdbc:oracle:thin:@");
jdbcURLString.append(props.getProperty("host"));
jdbcURLString.append(":");
jdbcURLString.append(props.getProperty("port"));
jdbcURLString.append(":");
jdbcURLString.append(props.getProperty("database"));
conn = DriverManager.getConnection(jdbcURLString.toString(), props
.getProperty("user"), props.getProperty("password"));
} catch (Exception e) {
e.printStackTrace();
} finally {
try {
fis.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return conn;
}
public int close() {
try {
newStatement.executeBatch();
conn.commit();
System.out.println("数据写入完毕");
this.newStatement.close();
this.statement.close();
this.conn.close();
return 1;
} catch (SQLException e) {
return 0;
}
}
}
package com.gaosheng.util.examples.xls;
import java.io.FileInputStream;
import java.io.IOException;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.List;
import java.util.Properties;
import com.gaosheng.util.xls.XxlsAbstract;
public class XxlsBig extends XxlsAbstract {
public static void main(String[] args) throws Exception {
XxlsBig howto = new XxlsBig("temp_table");
howto.processOneSheet("F:/new.xlsx",1);
howto.process("F:/new.xlsx");
howto.close();
}
public XxlsBig(String tableName) throws SQLException{
this.conn = getNew_Conn();
this.statement = conn.createStatement();
this.tableName = tableName;
}
private Connection conn = null;
private Statement statement = null;
private PreparedStatement newStatement = null;
private String tableName = "temp_table";
private boolean create = true;
public void optRows(int sheetIndex,int curRow, List<String> rowlist) throws SQLException {
if (sheetIndex == 0 && curRow == 0) {
StringBuffer preSql = new StringBuffer("insert into " + tableName
+ " values(");
StringBuffer table = new StringBuffer("create table " + tableName
+ "(");
int c = rowlist.size();
for (int i = 0; i < c; i++) {
preSql.append("?,");
table.append(rowlist.get(i));
table.append(" varchar2(100) ,");
}
table.deleteCharAt(table.length() - 1);
preSql.deleteCharAt(preSql.length() - 1);
table.append(")");
preSql.append(")");
if (create) {
statement = conn.createStatement();
try{
statement.execute("drop table "+tableName);
}catch(Exception e){
}finally{
System.out.println("表 "+tableName+" 删除成功");
}
if (!statement.execute(table.toString())) {
System.out.println("创建表 "+tableName+" 成功");
// return;
} else {
System.out.println("创建表 "+tableName+" 失败");
return;
}
}
conn.setAutoCommit(false);
newStatement = conn.prepareStatement(preSql.toString());
} else if(curRow>0) {
// 一般行
int col = rowlist.size();
for (int i = 0; i < col; i++) {
newStatement.setString(i + 1, rowlist.get(i).toString());
}
newStatement.addBatch();
if (curRow % 1000 == 0) {
newStatement.executeBatch();
conn.commit();
}
}
}
private static Connection getNew_Conn() {
Connection conn = null;
Properties props = new Properties();
FileInputStream fis = null;
try {
fis = new FileInputStream("D:/database.properties");
props.load(fis);
DriverManager.registerDriver(new oracle.jdbc.driver.OracleDriver());
// String jdbcURLString =
// "jdbc:oracle:thin:@192.168.0.28:1521:orcl";
StringBuffer jdbcURLString = new StringBuffer();
jdbcURLString.append("jdbc:oracle:thin:@");
jdbcURLString.append(props.getProperty("host"));
jdbcURLString.append(":");
jdbcURLString.append(props.getProperty("port"));
jdbcURLString.append(":");
jdbcURLString.append(props.getProperty("database"));
conn = DriverManager.getConnection(jdbcURLString.toString(), props
.getProperty("user"), props.getProperty("password"));
} catch (Exception e) {
e.printStackTrace();
} finally {
try {
fis.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return conn;
}
public int close() {
try {
newStatement.executeBatch();
conn.commit();
System.out.println("数据写入完毕");
this.newStatement.close();
this.statement.close();
this.conn.close();
return 1;
} catch (SQLException e) {
return 0;
}
}
}
继承类:XxlsPrint,作用:将数据输出到控制台
Java代码
package com.gaosheng.util.examples.xls;
import java.sql.SQLException;
import java.util.List;
import com.gaosheng.util.xls.XxlsAbstract;
public class XxlsPrint extends XxlsAbstract {
@Override
public void optRows(int sheetIndex,int curRow, List<String> rowlist) throws SQLException {
for (int i = 0; i < rowlist.size(); i++) {
System.out.print("'" + rowlist.get(i) + "',");
}
System.out.println();
}
public static void main(String[] args) throws Exception {
XxlsPrint howto = new XxlsPrint();
howto.processOneSheet("F:/new.xlsx",1);
// howto.processAllSheets("F:/new.xlsx");
}
}
源代码在附件中,还包含了说明文件、数据库配置文件、以及整合xls文件和xlsx文件读取的类:Xls2Do。
同之前的版本一样,大数据量文件的读取采用的是事件模型eventusermodel。usermodel模式需要将文件一次性全部读到内存中,07版的既然采用的存储模式是xml,解析用的DOM方式也是如此,这种模式操作简单,容易上手,但是对于大量数据占用的内存也是相当可观,在Eclipse中经常出现内存溢出。
下面就是采用eventusermodel对07excel文件读取。
同上篇,我将当前行的单元格数据存储到List中,抽象出 optRows 方法,该方法会在每行末尾时调用,方法参数为当前行索引curRow(int型)及存有行内单元格数据的List。继承类只需实现该行级方法即可。
经测试,对12万条数据,7M大小的文件也能正常运行。无需设置vm的内存空间。
excel读取采用的API为POI3.6,使用前先下载此包,若运行中出现其他依赖包不存在,请下载相应依赖包。
抽象类:XxlsAbstract ,作用:遍历excel文件,提供行级操作方法 optRows
package com.gaosheng.util.xls;
import java.io.InputStream;
import java.sql.SQLException;
import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
import org.apache.poi.xssf.eventusermodel.XSSFReader;
import org.apache.poi.xssf.model.SharedStringsTable;
import org.apache.poi.xssf.usermodel.XSSFRichTextString;
import org.apache.poi.openxml4j.opc.OPCPackage;
import org.xml.sax.Attributes;
import org.xml.sax.InputSource;
import org.xml.sax.SAXException;
import org.xml.sax.XMLReader;
import org.xml.sax.helpers.DefaultHandler;
import org.xml.sax.helpers.XMLReaderFactory;
/**
* XSSF and SAX (Event API)
*/
public abstract class XxlsAbstract extends DefaultHandler {
private SharedStringsTable sst;
private String lastContents;
private boolean nextIsString;
private int sheetIndex = -1;
private List<String> rowlist = new ArrayList<String>();
private int curRow = 0;
private int curCol = 0;
//excel记录行操作方法,以行索引和行元素列表为参数,对一行元素进行操作,元素为String类型
// public abstract void optRows(int curRow, List<String> rowlist) throws SQLException ;
//excel记录行操作方法,以sheet索引,行索引和行元素列表为参数,对sheet的一行元素进行操作,元素为String类型
public abstract void optRows(int sheetIndex,int curRow, List<String> rowlist) throws SQLException;
//只遍历一个sheet,其中sheetId为要遍历的sheet索引,从1开始,1-3
public void processOneSheet(String filename,int sheetId) throws Exception {
OPCPackage pkg = OPCPackage.open(filename);
XSSFReader r = new XSSFReader(pkg);
SharedStringsTable sst = r.getSharedStringsTable();
XMLReader parser = fetchSheetParser(sst);
// rId2 found by processing the Workbook
// 根据 rId# 或 rSheet# 查找sheet
InputStream sheet2 = r.getSheet("rId"+sheetId);
sheetIndex++;
InputSource sheetSource = new InputSource(sheet2);
parser.parse(sheetSource);
sheet2.close();
}
/**
* 遍历 excel 文件
*/
public void process(String filename) throws Exception {
OPCPackage pkg = OPCPackage.open(filename);
XSSFReader r = new XSSFReader(pkg);
SharedStringsTable sst = r.getSharedStringsTable();
XMLReader parser = fetchSheetParser(sst);
Iterator<InputStream> sheets = r.getSheetsData();
while (sheets.hasNext()) {
curRow = 0;
sheetIndex++;
InputStream sheet = sheets.next();
InputSource sheetSource = new InputSource(sheet);
parser.parse(sheetSource);
sheet.close();
}
}
public XMLReader fetchSheetParser(SharedStringsTable sst)
throws SAXException {
XMLReader parser = XMLReaderFactory
.createXMLReader("org.apache.xerces.parsers.SAXParser");
this.sst = sst;
parser.setContentHandler(this);
return parser;
}
public void startElement(String uri, String localName, String name,
Attributes attributes) throws SAXException {
// c => 单元格
if (name.equals("c")) {
// 如果下一个元素是 SST 的索引,则将nextIsString标记为true
String cellType = attributes.getValue("t");
if (cellType != null && cellType.equals("s")) {
nextIsString = true;
} else {
nextIsString = false;
}
}
// 置空
lastContents = "";
}
public void endElement(String uri, String localName, String name)
throws SAXException {
// 根据SST的索引值的到单元格的真正要存储的字符串
// 这时characters()方法可能会被调用多次
if (nextIsString) {
try {
int idx = Integer.parseInt(lastContents);
lastContents = new XSSFRichTextString(sst.getEntryAt(idx))
.toString();
} catch (Exception e) {
}
}
// v => 单元格的值,如果单元格是字符串则v标签的值为该字符串在SST中的索引
// 将单元格内容加入rowlist中,在这之前先去掉字符串前后的空白符
if (name.equals("v")) {
String value = lastContents.trim();
value = value.equals("")?" ":value;
rowlist.add(curCol, value);
curCol++;
}else {
//如果标签名称为 row ,这说明已到行尾,调用 optRows() 方法
if (name.equals("row")) {
try {
optRows(sheetIndex,curRow,rowlist);
} catch (SQLException e) {
e.printStackTrace();
}
rowlist.clear();
curRow++;
curCol = 0;
}
}
}
public void characters(char[] ch, int start, int length)
throws SAXException {
//得到单元格内容的值
lastContents += new String(ch, start, length);
}
}
package com.gaosheng.util.xls;
import java.io.InputStream;
import java.sql.SQLException;
import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
import org.apache.poi.xssf.eventusermodel.XSSFReader;
import org.apache.poi.xssf.model.SharedStringsTable;
import org.apache.poi.xssf.usermodel.XSSFRichTextString;
import org.apache.poi.openxml4j.opc.OPCPackage;
import org.xml.sax.Attributes;
import org.xml.sax.InputSource;
import org.xml.sax.SAXException;
import org.xml.sax.XMLReader;
import org.xml.sax.helpers.DefaultHandler;
import org.xml.sax.helpers.XMLReaderFactory;
/**
* XSSF and SAX (Event API)
*/
public abstract class XxlsAbstract extends DefaultHandler {
private SharedStringsTable sst;
private String lastContents;
private boolean nextIsString;
private int sheetIndex = -1;
private List<String> rowlist = new ArrayList<String>();
private int curRow = 0;
private int curCol = 0;
//excel记录行操作方法,以行索引和行元素列表为参数,对一行元素进行操作,元素为String类型
// public abstract void optRows(int curRow, List<String> rowlist) throws SQLException ;
//excel记录行操作方法,以sheet索引,行索引和行元素列表为参数,对sheet的一行元素进行操作,元素为String类型
public abstract void optRows(int sheetIndex,int curRow, List<String> rowlist) throws SQLException;
//只遍历一个sheet,其中sheetId为要遍历的sheet索引,从1开始,1-3
public void processOneSheet(String filename,int sheetId) throws Exception {
OPCPackage pkg = OPCPackage.open(filename);
XSSFReader r = new XSSFReader(pkg);
SharedStringsTable sst = r.getSharedStringsTable();
XMLReader parser = fetchSheetParser(sst);
// rId2 found by processing the Workbook
// 根据 rId# 或 rSheet# 查找sheet
InputStream sheet2 = r.getSheet("rId"+sheetId);
sheetIndex++;
InputSource sheetSource = new InputSource(sheet2);
parser.parse(sheetSource);
sheet2.close();
}
/**
* 遍历 excel 文件
*/
public void process(String filename) throws Exception {
OPCPackage pkg = OPCPackage.open(filename);
XSSFReader r = new XSSFReader(pkg);
SharedStringsTable sst = r.getSharedStringsTable();
XMLReader parser = fetchSheetParser(sst);
Iterator<InputStream> sheets = r.getSheetsData();
while (sheets.hasNext()) {
curRow = 0;
sheetIndex++;
InputStream sheet = sheets.next();
InputSource sheetSource = new InputSource(sheet);
parser.parse(sheetSource);
sheet.close();
}
}
public XMLReader fetchSheetParser(SharedStringsTable sst)
throws SAXException {
XMLReader parser = XMLReaderFactory
.createXMLReader("org.apache.xerces.parsers.SAXParser");
this.sst = sst;
parser.setContentHandler(this);
return parser;
}
public void startElement(String uri, String localName, String name,
Attributes attributes) throws SAXException {
// c => 单元格
if (name.equals("c")) {
// 如果下一个元素是 SST 的索引,则将nextIsString标记为true
String cellType = attributes.getValue("t");
if (cellType != null && cellType.equals("s")) {
nextIsString = true;
} else {
nextIsString = false;
}
}
// 置空
lastContents = "";
}
public void endElement(String uri, String localName, String name)
throws SAXException {
// 根据SST的索引值的到单元格的真正要存储的字符串
// 这时characters()方法可能会被调用多次
if (nextIsString) {
try {
int idx = Integer.parseInt(lastContents);
lastContents = new XSSFRichTextString(sst.getEntryAt(idx))
.toString();
} catch (Exception e) {
}
}
// v => 单元格的值,如果单元格是字符串则v标签的值为该字符串在SST中的索引
// 将单元格内容加入rowlist中,在这之前先去掉字符串前后的空白符
if (name.equals("v")) {
String value = lastContents.trim();
value = value.equals("")?" ":value;
rowlist.add(curCol, value);
curCol++;
}else {
//如果标签名称为 row ,这说明已到行尾,调用 optRows() 方法
if (name.equals("row")) {
try {
optRows(sheetIndex,curRow,rowlist);
} catch (SQLException e) {
e.printStackTrace();
}
rowlist.clear();
curRow++;
curCol = 0;
}
}
}
public void characters(char[] ch, int start, int length)
throws SAXException {
//得到单元格内容的值
lastContents += new String(ch, start, length);
}
}
继承类:XxlsBig,作用:将数据转出到数据库临时表
Java代码
package com.gaosheng.util.examples.xls;
import java.io.FileInputStream;
import java.io.IOException;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.List;
import java.util.Properties;
import com.gaosheng.util.xls.XxlsAbstract;
public class XxlsBig extends XxlsAbstract {
public static void main(String[] args) throws Exception {
XxlsBig howto = new XxlsBig("temp_table");
howto.processOneSheet("F:/new.xlsx",1);
howto.process("F:/new.xlsx");
howto.close();
}
public XxlsBig(String tableName) throws SQLException{
this.conn = getNew_Conn();
this.statement = conn.createStatement();
this.tableName = tableName;
}
private Connection conn = null;
private Statement statement = null;
private PreparedStatement newStatement = null;
private String tableName = "temp_table";
private boolean create = true;
public void optRows(int sheetIndex,int curRow, List<String> rowlist) throws SQLException {
if (sheetIndex == 0 && curRow == 0) {
StringBuffer preSql = new StringBuffer("insert into " + tableName
+ " values(");
StringBuffer table = new StringBuffer("create table " + tableName
+ "(");
int c = rowlist.size();
for (int i = 0; i < c; i++) {
preSql.append("?,");
table.append(rowlist.get(i));
table.append(" varchar2(100) ,");
}
table.deleteCharAt(table.length() - 1);
preSql.deleteCharAt(preSql.length() - 1);
table.append(")");
preSql.append(")");
if (create) {
statement = conn.createStatement();
try{
statement.execute("drop table "+tableName);
}catch(Exception e){
}finally{
System.out.println("表 "+tableName+" 删除成功");
}
if (!statement.execute(table.toString())) {
System.out.println("创建表 "+tableName+" 成功");
// return;
} else {
System.out.println("创建表 "+tableName+" 失败");
return;
}
}
conn.setAutoCommit(false);
newStatement = conn.prepareStatement(preSql.toString());
} else if(curRow>0) {
// 一般行
int col = rowlist.size();
for (int i = 0; i < col; i++) {
newStatement.setString(i + 1, rowlist.get(i).toString());
}
newStatement.addBatch();
if (curRow % 1000 == 0) {
newStatement.executeBatch();
conn.commit();
}
}
}
private static Connection getNew_Conn() {
Connection conn = null;
Properties props = new Properties();
FileInputStream fis = null;
try {
fis = new FileInputStream("D:/database.properties");
props.load(fis);
DriverManager.registerDriver(new oracle.jdbc.driver.OracleDriver());
// String jdbcURLString =
// "jdbc:oracle:thin:@192.168.0.28:1521:orcl";
StringBuffer jdbcURLString = new StringBuffer();
jdbcURLString.append("jdbc:oracle:thin:@");
jdbcURLString.append(props.getProperty("host"));
jdbcURLString.append(":");
jdbcURLString.append(props.getProperty("port"));
jdbcURLString.append(":");
jdbcURLString.append(props.getProperty("database"));
conn = DriverManager.getConnection(jdbcURLString.toString(), props
.getProperty("user"), props.getProperty("password"));
} catch (Exception e) {
e.printStackTrace();
} finally {
try {
fis.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return conn;
}
public int close() {
try {
newStatement.executeBatch();
conn.commit();
System.out.println("数据写入完毕");
this.newStatement.close();
this.statement.close();
this.conn.close();
return 1;
} catch (SQLException e) {
return 0;
}
}
}
package com.gaosheng.util.examples.xls;
import java.io.FileInputStream;
import java.io.IOException;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.List;
import java.util.Properties;
import com.gaosheng.util.xls.XxlsAbstract;
public class XxlsBig extends XxlsAbstract {
public static void main(String[] args) throws Exception {
XxlsBig howto = new XxlsBig("temp_table");
howto.processOneSheet("F:/new.xlsx",1);
howto.process("F:/new.xlsx");
howto.close();
}
public XxlsBig(String tableName) throws SQLException{
this.conn = getNew_Conn();
this.statement = conn.createStatement();
this.tableName = tableName;
}
private Connection conn = null;
private Statement statement = null;
private PreparedStatement newStatement = null;
private String tableName = "temp_table";
private boolean create = true;
public void optRows(int sheetIndex,int curRow, List<String> rowlist) throws SQLException {
if (sheetIndex == 0 && curRow == 0) {
StringBuffer preSql = new StringBuffer("insert into " + tableName
+ " values(");
StringBuffer table = new StringBuffer("create table " + tableName
+ "(");
int c = rowlist.size();
for (int i = 0; i < c; i++) {
preSql.append("?,");
table.append(rowlist.get(i));
table.append(" varchar2(100) ,");
}
table.deleteCharAt(table.length() - 1);
preSql.deleteCharAt(preSql.length() - 1);
table.append(")");
preSql.append(")");
if (create) {
statement = conn.createStatement();
try{
statement.execute("drop table "+tableName);
}catch(Exception e){
}finally{
System.out.println("表 "+tableName+" 删除成功");
}
if (!statement.execute(table.toString())) {
System.out.println("创建表 "+tableName+" 成功");
// return;
} else {
System.out.println("创建表 "+tableName+" 失败");
return;
}
}
conn.setAutoCommit(false);
newStatement = conn.prepareStatement(preSql.toString());
} else if(curRow>0) {
// 一般行
int col = rowlist.size();
for (int i = 0; i < col; i++) {
newStatement.setString(i + 1, rowlist.get(i).toString());
}
newStatement.addBatch();
if (curRow % 1000 == 0) {
newStatement.executeBatch();
conn.commit();
}
}
}
private static Connection getNew_Conn() {
Connection conn = null;
Properties props = new Properties();
FileInputStream fis = null;
try {
fis = new FileInputStream("D:/database.properties");
props.load(fis);
DriverManager.registerDriver(new oracle.jdbc.driver.OracleDriver());
// String jdbcURLString =
// "jdbc:oracle:thin:@192.168.0.28:1521:orcl";
StringBuffer jdbcURLString = new StringBuffer();
jdbcURLString.append("jdbc:oracle:thin:@");
jdbcURLString.append(props.getProperty("host"));
jdbcURLString.append(":");
jdbcURLString.append(props.getProperty("port"));
jdbcURLString.append(":");
jdbcURLString.append(props.getProperty("database"));
conn = DriverManager.getConnection(jdbcURLString.toString(), props
.getProperty("user"), props.getProperty("password"));
} catch (Exception e) {
e.printStackTrace();
} finally {
try {
fis.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return conn;
}
public int close() {
try {
newStatement.executeBatch();
conn.commit();
System.out.println("数据写入完毕");
this.newStatement.close();
this.statement.close();
this.conn.close();
return 1;
} catch (SQLException e) {
return 0;
}
}
}
继承类:XxlsPrint,作用:将数据输出到控制台
Java代码
package com.gaosheng.util.examples.xls;
import java.sql.SQLException;
import java.util.List;
import com.gaosheng.util.xls.XxlsAbstract;
public class XxlsPrint extends XxlsAbstract {
@Override
public void optRows(int sheetIndex,int curRow, List<String> rowlist) throws SQLException {
for (int i = 0; i < rowlist.size(); i++) {
System.out.print("'" + rowlist.get(i) + "',");
}
System.out.println();
}
public static void main(String[] args) throws Exception {
XxlsPrint howto = new XxlsPrint();
howto.processOneSheet("F:/new.xlsx",1);
// howto.processAllSheets("F:/new.xlsx");
}
}
源代码在附件中,还包含了说明文件、数据库配置文件、以及整合xls文件和xlsx文件读取的类:Xls2Do。
- src.rar (9.7 KB)
- 下载次数: 10
相关推荐
java 使用 poi 解析导入大数据量(几万数据量+)时,报出OOM。这是使用POI 第二种处理方法,解决大数据量导入内存溢出问题,并提升效率
NULL 博文链接:https://domcafe.iteye.com/blog/1472399
js前端处理excel数据导入,支持大数据处理,自定义分片上传,加载动画
其次,在实验过程中,大数据量的导出很容易引发内存溢出,调整JVM的内存大小治标不治本。很多人建议保存为.CSV格式的文件。不过,.CSV方式导出也存在问题:首先,如果用excel来打开csv,超过65536行的数据都会看不见...
usermodel模式对excel操作前需要将文件全部转入内存,对较大文件来说内存开销很大。但是其使用简单。 eventusermodel模式采用事件模型,对文件边读取边处理,内存消耗较低,效率高,因为不用等待文件全部装入内存。...
宏软Excel助手是一款强大的Excel数据处理辅助软件,可以帮助用户对大量的Excel进行处理,支持报表功能,兼容MFC数据库,可以与VF数据库对接,并集成了搜索精灵、区位码精灵、万年历等实用工具,非常适合学校、公司、...
从http://download.csdn.net/detail/whatismvc/3696185 和http://download.csdn.net/detail/whatismvc/3694229 下载的, 处理大数据量的Excel 2007文件不内存溢出,我试过的最大数据是 26000行,222列的xlsx。
java解决大批量数据导出Excel产生内存溢出的方案
excle大量数据读取,10万条数据没有问题,一般的方法很可能出现内存溢出
C# datatable直接导出数据到Excel,(数据量百万级只需3秒)
由于项目需要对大量Excel数据进行输入输出处理,在使用JXL,POI后发现很容易出现OOM,最后在网上找到阿里的开源项目EasyExcel能很快速的读取写入超大Excel文件。经过大量的调试优化,现通过JAVA生成104万行20列的...
[Excel数据处理与分析实战技巧精粹]专业版是市场上最为强大便捷的Excel比较工具。它为工作中经常需要进行数据比较的用户提供了完美的解决方案。无论你的数据是存放在Excel文件,还是存放在文本文件,或者存放在...
batchxls破解版是一款功能强大的Excel文档批量处理辅助工具。你可以永久免费使用它,它是一款简单好用的可以对已有的Excel文件进行多样化处理的工具,这款软件功能强大,非常便捷可以一次将多个Excel文档中的指定...
主要为大家详细介绍了Java实现excel大数据量导入,文中示例代码介绍的非常详细,具有一定的参考价值,感兴趣的小伙伴们可以参考一下
前端Excel大文件file slice分片,md5校验文件完整性并作文件标识记录写入数据库,支持断点...文件上传完毕后,使用EasyExcel读取文件流,处理Excel数据写入数据库中,可处理百万级数据。项目完整,连接数据库即可运行。
从http://download.csdn.net/detail/whatismvc/3696185 ...加上自己整合的 eventusermodel模式采用事件模型,对文件边读取边处理,内存消耗较低,效率高,因为不用等待文件全部装入内存。但使用较复杂。
中文名: 别说你懂Excel:500招玩转Excel表格与数据处理(附完整光盘数据) 作者: 前沿文化图书fenlei: 软件 资源格式: PDF 版本: 扫描版 出版社: 科学出版社书号: 9787030371782发行时间: 2013年05月 地区: 大陆 语言:...
该代码可以处理100万数据量的excel文件,xlsx文件数据量太大,用普通的读法会报内存溢出错误,所以用官网提供的方法,一条一条的读取大excel文件,本例子从这点出发,组装excel里读取的单条数据为list,在根据需求...
巧妙应用Excel处理测量数据