[xwiki-users] MySQL storage using MyISAM engine.
Hi, I have an instance of XWiki Enterprise v2.1.1 deployed using a MySQL database. I've noticed that the engine used in MySQL is MyISAM and was wondering if there was a particular reason why InnoDB was not the engine of choice. Won't the use of MyISAM prove troublesome if lots of attachments are added to the Wiki? MyISAM uses a single file for all data in a table, and thus all attachements being stored in the same table, this MyISAM file might grow huge, which is usually not something suitable. Any past experience with attachement heavy Wiki instances using MySQL for storage? Any plan of using InnoDB in the near future? Thanks, Mathias.
On Thu, Jan 14, 2010 at 13:16, Mathias Herberts <[email protected]> wrote:
Hi,
I have an instance of XWiki Enterprise v2.1.1 deployed using a MySQL database. I've noticed that the engine used in MySQL is MyISAM and was wondering if there was a particular reason why InnoDB was not the engine of choice.
Won't the use of MyISAM prove troublesome if lots of attachments are added to the Wiki? MyISAM uses a single file for all data in a table, and thus all attachements being stored in the same table, this MyISAM file might grow huge, which is usually not something suitable.
Any past experience with attachement heavy Wiki instances using MySQL for storage? Any plan of using InnoDB in the near future?
XWiki does not choose any specific MySQL engine AFAIK. it just ask for a table to the server and the server choose he default one. Or maybe hibernate generate initialization script with MyISAM explicitly but it would be weird. Maybe you can choose the MySQL engine in the hibernate configuration file to make sure it's using InnoDB.
Thanks,
Mathias. _______________________________________________ users mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/users
-- Thomas Mortagne
On Thu, Jan 14, 2010 at 13:29, Thomas Mortagne <[email protected]> wrote:
On Thu, Jan 14, 2010 at 13:16, Mathias Herberts <[email protected]> wrote:
Hi,
I have an instance of XWiki Enterprise v2.1.1 deployed using a MySQL database. I've noticed that the engine used in MySQL is MyISAM and was wondering if there was a particular reason why InnoDB was not the engine of choice.
Ok, I checked Hibernate and found that by using dialect 'org.hibernate.dialect.MySQL5InnoDBDialect' you could switch to InnoDB tables. But now there seems to be a problem with at least one XWiki table, namely xwikicomments: create table xwikicomments (XWC_ID integer not null, XWC_AUTHOR varchar(255), XWC_HIGHLIGHT varchar(60000), XWC_COMMENT varchar(60000), XWP_REPLYTO integer, XWP_DATE datetime, primary key (XWC_ID)) ENGINE=InnoDB] com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Row size too large. The maximum row size for the used table type, not counting BLOBs, is 65535. You have to change some columns to TEXT or BLOBs cf http://dev.mysql.com/doc/refman/5.0/en/innodb-restrictions.html so this is not as trivial as expected :-( Mathias.
On 14/01/10 11:29 PM, Thomas Mortagne wrote:
On Thu, Jan 14, 2010 at 13:16, Mathias Herberts <[email protected]> wrote:
Hi,
I have an instance of XWiki Enterprise v2.1.1 deployed using a MySQL database. I've noticed that the engine used in MySQL is MyISAM and was wondering if there was a particular reason why InnoDB was not the engine of choice.
Won't the use of MyISAM prove troublesome if lots of attachments are added to the Wiki? MyISAM uses a single file for all data in a table, and thus all attachements being stored in the same table, this MyISAM file might grow huge, which is usually not something suitable.
Any past experience with attachement heavy Wiki instances using MySQL for storage? Any plan of using InnoDB in the near future?
XWiki does not choose any specific MySQL engine AFAIK. it just ask for a table to the server and the server choose he default one. Or maybe hibernate generate initialization script with MyISAM explicitly but it would be weird. Maybe you can choose the MySQL engine in the hibernate configuration file to make sure it's using InnoDB.
MyISAM is the default storage engine for MySQL. You can set it to something different if you wish, within the mysql server: http://dev.mysql.com/doc/refman/5.1/en/storage-engine-setting.html
Thanks,
Mathias. _______________________________________________ users mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/users
MyISAM is the default storage engine for MySQL. You can set it to something different if you wish, within the mysql server: http://dev.mysql.com/doc/refman/5.1/en/storage-engine-setting.html
The way some fields are specified in the hbm files (with type string), constraints the MySQL engine to be MyISAM as InnoDB cannot have rows of more than 65535 bytes, so using InnoDB would imply modifying XWiki and not just changing its configuration. Are there any plans in supporting JCR (JSR-170) for attachement storage in future versions of XWiki? Mathias.
Hi, On Jan 14, 2010, at 2:38 PM, Mathias Herberts wrote:
MyISAM is the default storage engine for MySQL. You can set it to something different if you wish, within the mysql server: http://dev.mysql.com/doc/refman/5.1/en/storage-engine-setting.html
The way some fields are specified in the hbm files (with type string), constraints the MySQL engine to be MyISAM as InnoDB cannot have rows of more than 65535 bytes, so using InnoDB would imply modifying XWiki and not just changing its configuration.
Are there any plans in supporting JCR (JSR-170) for attachement storage in future versions of XWiki?
Yes. Thanks -Vincent
Given that the current implementation is innodb for now, one possibility to consider, at least on some platforms is to use large pages for better OS performance. (If anybody tries this, would love to hear your results.). Wouldn't large pages solve some of the efficiency issues for documents using lots of attachments? http://lists.xwiki.org/pipermail/devs/2009-August/013849.html
Has anybody tried enabling "large page support" in Mysql for Xwiki?
Doesn't Xwiki make use of Large pages for document attachments and other
storage, and wouldn't this be useful for high-volume Xwiki based systems? (Fedora Linux has this feature enabled by default.)
The improvement? "Applications that perform a lot of memory accesses may
obtain performance improvements by using large pages due to reduced Translation Lookaside Buffer (TLB) misses."
Is this worth doing? Would it improve performance (or are bottlenecks
elsewhere that make this tweak irrelevant)?
............
http://dev.mysql.com/doc/refman/5.0/en/innodb-configuration.html
--> http://dev.mysql.com/doc/refman/5.0/en/large-page-support.html
7.5.9. Enabling Large Page Support
Some hardware/operating system architectures support memory pages greater than the default (usually 4KB). The actual implementation of this support depends on the underlying hardware and operating system. Applications that perform a lot of memory accesses may obtain performance improvements by using large pages due to reduced Translation Lookaside Buffer (TLB) misses.
In MySQL, large pages can be used by InnoDB, to allocate memory for its buffer pool and additional memory pool.
Currently, MySQL supports only the Linux implementation of large page support (which is called HugeTLB in Linux).
Before large pages can be used on Linux, the kernel must be enabled to support them and it is necessary to configure the HugeTLB memory pool. For reference, the HugeTBL API is documented in the Documentation/vm/hugetlbpage.txt file of your Linux sources.
The kernel for some recent systems such as Red Hat Enterprise Linux appear to have the large pages feature enabled by default. To check whether this is true for your kernel, use the following command and look for output lines containing “huge”:
Since it's available in RHEL, it's also available in fedora12<http://fedoraproject.org/> mysql , but not used by Xwikis running on this mysql currently: ROOT-31-~> cat /proc/meminfo | grep -i huge HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB -- Niels http://nielsmayer.com
BTW a few more interesting links and excepts on this subject. Seems like if one can devote a certain amount of RAM to Mysql, this could potentially be a performace win for a larger Xwiki install: http://www.basysadmin.com/procedures.html
Configure mysql (InnoDB) to use hugepages. Hugepages are not swappable. [Ref]<http://dev.mysql.com/doc/refman/5.1-maria/en/large-page-support.html> [Ref]<http://www.cyberciti.biz/tips/linux-hugetlbfs-and-mysql-performance.html>
http://www.cyberciti.biz/files/linux-kernel/Documentation/vm/hugetlbpage.txt http://docs.sun.com/source/mysql-refman-5.0/optimization.html http://www.cyberciti.biz/tips/linux-hugetlbfs-and-mysql-performance.html A pplications that perform a lot of memory accesses (several GBs) may obtain performance improvements by using large pages due to reduced Translation Lookaside Buffer (TLB) misses. HugeTLBfs is memory management feature offered in Linux kernel, which is valuable for applications that use a large virtual address space. It is especially useful for database applications such as MySQL, Oracle and others. Other server software that uses the prefork or similar (e.g. Apache web server) model will also benefit. The CPU's Translation Lookaside Buffer (TLB) is a small cache used for storing virtual-to-physical mapping information. By using the TLB, a translation can be performed without referencing the in-memory page table entry that maps the virtual address. However, to keep translations as fast as possible, the TLB is usually small. It is not uncommon for large memory applications to exceed the mapping capacity of the TLB. Users can use the huge page support in Linux kernel by either using the mmap system call or standard SYSv shared memory system calls (shmget, shmat). Only selected hardware and operating system support memory pages greater than the default 4KB. The following configuration tested on RHEL 5.3 64 bit using a stock kernel with tons of RAM and multiple CPUs. -- Niels http://nielsmayer.com
participants (5)
-
Mathias Herberts -
Niels Mayer -
Nigel Smith -
Thomas Mortagne -
Vincent Massol