Quantcast
Channel: MySQL Support Blogs
Viewing all 352 articles
Browse latest View live

MySQL Connect slides

$
0
0

Thank you for attending MySQL Connect 2013.

The Saturday session 50 Tips for Boosting MySQL Performance [CON2655] was sold out. It shows the interest in practical recipes to solve performance problems. Maybe the topic of a book as suggested by the audience ?

On Monday, the tutorial Enhancing Productivity with MySQL 5.6 New Features [TUT8131] was less crowded due to MySQL 5.6 Replication Tips and Tricks [TUT8133]  happening at the same time, again tips and tricks are more popular :-) .

Here are the slides :

The ppt files are here (additional notes) :

Feel free to comment, report problems or ask questions if unclear !

 


Pre-5.0.30 ADDTIME()

$
0
0

For when you want to add times greater than 839 hours.

CREATE FUNCTION ADDTIME_OLD(d datetime, t varchar(12))
RETURNS datetime
DETERMINISTIC
RETURN ADDTIME(
d + INTERVAL SUBSTRING_INDEX(t, ':', 1) HOUR,
CONCAT(IF(LEFT(t, 1) = '-', '-', ''), '00:', SUBSTRING_INDEX(t, ':', -2))
);

(Almost the same. I'm not going to figure out how to do the "you can use any delimiter you want" that MySQL supports.)

Poor man’s Online Optimize in 5.6

$
0
0

Table space fragmentation has generally 2 origins :

  • File System fragmentation : the data file is spread physically on many non contiguous locations on the disk.
  • Internal Fragmentation : the data and index pages have “holes” : this happens when  rows are deleted or updated, especially at random.

As a result, performance is affected by table space fragmentation. Data typically takes more space on disk and in memory. The disk is more busy than it should.

File System fragmentation can be detected using the filefrag command on Linux (and similar on different OS). When using MyISAM, MYI files are usually very fragmented on the FS, much more than the MYD files.

ls -al frag
total 883304
drwx------  2 aadant common      4096 Sep 30 18:41 .
drwxr-xr-x 17 aadant common      4096 Sep 30 18:59 ..
-rw-rw----  1 aadant common        65 Sep 30 18:40 db.opt
-rw-rw----  1 aadant common      8608 Sep 30 18:41 t.frm
-rw-rw----  1 aadant common 551944192 Sep 30 19:41 t.MYD
-rw-rw----  1 aadant common 426150912 Sep 30 19:41 t.MYI
filefrag frag/t.MYD
frag/t.MYD: 23 extents found
filefrag frag/t.MYI
frag/t.MYI: 4949 extents found

To measure internal InnoDB fragmentation, there is no general formula. It is possible to compute the theoretical size and compare it to the actual size on disk. It only works on fixed length data types. I have this trick : it is mentioned in 50 Tips for Boosting MySQL Performance [CON2655]. The idea is to compare the fragmented table and a much smaller table having the same table definition and sample data.
It works well when the average row length can be estimated from the sample data even for variable length rows.

create table t_defrag like t; 
insert into t_defrag select * from t limit 20000;

Here 20000 rows 15-20 Mb on disk gives a good idea. Inserting the sample data the same empty table actually defragments the sample, since it contains non empty rows.

The table is likely internally fragmented

if Avg_row_length(t) > Avg_row_length(t_defrag)

where Avg_row_length comes from show table status.

Example : some random data is created, uniformly distributed random row length : 1M rows, 640Mb. Internal fragmentation is created by deleting the even rows (one every two rows).

drop database test_case2;
create database test_case2;
use test_case2;
set global query_cache_size = 0;
set global innodb_flush_log_at_trx_commit = 2;
drop table if exists t1;
create table t1(
id int primary key auto_increment,
k  varchar(50),
value varchar(1000), unique key(k)) engine=InnoDB;
set @t1 =(select now());
set @i := 1;
insert into t1(k,value) values(@i:=@i+1,repeat('a',rand()*1000)); 
insert into t1(k,value) values(@i:=@i+1,repeat('a',rand()*1000)); 
replace into t1(k,value) select @i:=@i+1,repeat('a',rand()*1000) from t1 t1, t1 t2, t1 t3, t1 t4, t1 t5, t1 t6, t1 t7, t1 t8, t1 t9, t1 t10, t1 t11, t1 t12, t1 t13, t1 t14, t1 t15, t1 t16, t1 t17,
t1 t18, t1 t19, t1 t20;
set @t2 =(select now());
select @@version,timediff(@t2,@t1) duration;

analyze table t1\G
show table status like 't1'\G
delete from t1 where mod(id,2) = 0; 
show table status like 't1'\G
drop table if exists t1_defrag;
create table t1_defrag like t1; 
insert into t1_defrag select * from t1 limit 20000;
analyze table t1\G
analyze table t1_defrag\G
show table status like 't1'\G
show table status like 't1_defrag'\G

mysql> show table status like 't1'\G
*************************** 1. row ***************************
           Name: t1
         Engine: InnoDB
        Version: 10
     Row_format: Compact
           Rows: 476676
 Avg_row_length: 1293
    Data_length: 616562688
Max_data_length: 0
   Index_length: 37257216
      Data_free: 5242880
 Auto_increment: 1114098
    Create_time: 2013-09-30 15:19:01
    Update_time: NULL
     Check_time: NULL
      Collation: latin1_swedish_ci
       Checksum: NULL
 Create_options:
        Comment:
1 row in set (0.01 sec)

mysql> show table status like 't1_defrag'\G
*************************** 1. row ***************************
           Name: t1_defrag
         Engine: InnoDB
        Version: 10
     Row_format: Compact
           Rows: 19773
 Avg_row_length: 610
    Data_length: 12075008
Max_data_length: 0
   Index_length: 1589248
      Data_free: 4194304
 Auto_increment: 40000
    Create_time: 2013-09-30 15:23:38
    Update_time: NULL
     Check_time: NULL
      Collation: latin1_swedish_ci
       Checksum: NULL
 Create_options:
        Comment:
1 row in set (0.01 sec)

ls -al data/test_case2/t1*
-rw-rw---- 1 aadant common      8612 Sep 30 20:39 data/test_case2/t1_defrag.frm
-rw-rw---- 1 aadant common  22020096 Sep 30 20:39 data/test_case2/t1_defrag.ibd
-rw-rw---- 1 aadant common      8612 Sep 30 20:40 data/test_case2/t1.frm
-rw-rw---- 1 aadant common 671088640 Sep 30 20:44 data/test_case2/t1.ibd

As you can see, the table t1 is fragmented since its average row length is 2 times larger than t1_defrag.

Fortunately, there is a command that fixes both FS and internal fragmentation on most storage engines :

optimize table t1;

For InnoDB, optimize does rebuild the table and analyze it. So this command also works :

alter table t1 engine=InnoDB;

mysql> alter table t1 engine=InnoDB;
Query OK, 524289 rows affected (5 min 58.99 sec)
Records: 524289  Duplicates: 0  Warnings: 0

mysql> show table status like 't1'\G
*************************** 1. row ***************************
           Name: t1
         Engine: InnoDB
        Version: 10
     Row_format: Compact
           Rows: 500516
 Avg_row_length: 611
    Data_length: 305971200
Max_data_length: 0
   Index_length: 18399232
      Data_free: 5242880
 Auto_increment: 1048578
    Create_time: 2013-09-30 19:53:47
    Update_time: NULL
     Check_time: NULL
      Collation: latin1_swedish_ci
       Checksum: NULL
 Create_options:
        Comment:
1 row in set (0.26 sec)

After the table rebuild, the average row length is back to normal and the table size is minimal on disk, here 2 times smaller, 324Mb.

ls -al data/test_case2/t1*
-rw-rw---- 1 aadant common      8612 Sep 30 20:35 data/test_case2/t1.frm
-rw-rw---- 1 aadant common 339738624 Sep 30 20:37 data/test_case2/t1.ibd

Note that the optimize and alter table engine=InnoDB commands take an exclusive lock on the table. It means that if you use these commands your application : reads, writes, other DDL will be blocked during the table rebuild. So most of the time these commands were run in maintenance windows. Or using special tricks such as pt-online-schema-change or Online Schema Change at Facebook.

In MySQL 5.6, this is no longer required in most cases thanks to InnoDB online DDL. Online DDL is a major 5.6 feature. Even though optimize table is not an online operation, in practice, you can use this online DDL :

alter table t1 row_format = <row format>;

where <row_format> is given by show table status. Usually :

alter table t1 row_format = Compact;

Make sure :

Check also : innodb_sort_buffer_size that can speed up online DDL.

 alter table t1 row_format=Compact;
Query OK, 0 rows affected (5 min 26.53 sec)
Records: 0  Duplicates: 0  Warnings: 0

mysql> show table status like 't1'\G
*************************** 1. row ***************************
           Name: t1
         Engine: InnoDB
        Version: 10
     Row_format: Compact
           Rows: 502312
 Avg_row_length: 613
    Data_length: 308068352
Max_data_length: 0
   Index_length: 9977856
      Data_free: 0
 Auto_increment: 1114098
    Create_time: 2013-09-30 20:53:41
    Update_time: NULL
     Check_time: NULL
      Collation: latin1_swedish_ci
       Checksum: NULL
 Create_options: row_format=COMPACT
        Comment:
1 row in set (0.02 sec)

The disk usage is even slightly smaller, 320Mb, than the classical rebuild and it is also slightly faster.

ls -al data/test_case2/t1*
-rw-rw---- 1 aadant common      8612 Sep 30 20:51 data/test_case2/t1.frm
-rw-rw---- 1 aadant common 318767104 Sep 30 20:53 data/test_case2/t1.ibd

With innodb_sort_buffer_size= 32M, the alter is faster :

alter table t1 row_format=Compact;
Query OK, 0 rows affected (2 min 51.96 sec)
Records: 0  Duplicates: 0  Warnings: 0

 

JSON UDFs first feedback

$
0
0

Yesterday Ulf Wendel created great blog post with deep analysis of JSON UDF functions which I presented at MySQL Connect at September, 21.

Ulf found few bugs in these functions, I reported them all at bugs.mysql.com. You can find numbers in my comment to his blog post. But he also raised concerns, which can not be considered pure bugs, rather feature requests, or even design especiallities.

* First concern, of course, is the documentation. Ulf writes: "Here’s what the README, the only documentation available apart from the *.c[omment]/*.h[elp] files".

I agree single README file is not enough, but this is still Labs project for which I can not abuse MySQL documentation team for making proper documentation for me. But you still can find more information, than single README file. And these are slides from MySQL Connect, which are available on the conference website (they published them today!) at https://oracleus.activeevents.com/2013/connect/fileDownload/session/470E8AA1C46CAA3A9ACFBDB7653FC313/CON3130_Smirnova.pdf as well as on my own website at http://www.microbecal.com/ftp_load/JSON_UDFs.odp or http://www.microbecal.com/ftp_load/JSON_UDFs.pdf I strongly recommend you to download these slides, because this is the best user manual for JSON functions which exists so far.

But I also want to hear your suggestions about where I can put the complete user manual. I was going to post updates to my blog and update README file. Is this sufficient? Or do you think would be better if I put the documentation at my own website? Or pack it as a separate *-doc package at labs.mysql.com? Any other ideas? If you also miss the documentation, please comment.

* Another request which I have for you is about parameter syntax. Ulf writes: "A function that works “by key” on a subset of a JSON document, usually has the prototype: JSON_SOMETHING(string json_document, string key [, string key...], other_parameter). The first argument passed to the functions is always the JSON document to work on. It follows a variable number of arguments that describe on which subpart of the entire document the function shall work. Having this string key [, string key...] arguments in the middle of the function signature is a bit odd. Some programming languages may forbid this for style considerations. However, depending on the function, further arguments may follow, such as the value to search for or to add."

We discussed this syntax internally before I started implementing the functions and get to this style. But I understand what some people could prefer different syntax. If you such a person, please, file a feature request. At this stage it is easy to rewrite functions, so they support better style of user arguments. I will think some time about it and will post a poll to my blog with proposals which I think can be better than current syntax. Also it is possible to have different set of functions with the same functionality, but which support different style of arguments. Only requirement for this feature is users suggestions and votes. Therefore, please suggest and vote!

* Another feature for which I need your suggestions is "speed vs validity checks". I again will cite Ulf: "Taken from the README. For most functions manipulating or generating JSON documents is true: Warning! This version does not check whole document for validity. Hey, it is a labs.mysql.com pre-production release :-)."

But this design was done not because these functions are at the Labs and not because this is alpha version. We discussed this functionality internally, but decided to don't implement validity checks to speed up functions. If you think we did wrong, please, open a feature request at bugs.mysql.com about *safe* version of the functions. I also think that having two sets of "safe" and "fast" functions can also work. But I need to hear your opinions before implementing this feature.

* Another feature for which I want to count user votes is search abilities. Currently, function JSON_SEARCH uses only exact match. There is no support for wildcards, case insensitive search or full-text search. I got complains about such a limitation not only from Ulf, but from users at MySQL Connect too. Therefore I created two feature requests: http://bugs.mysql.com/bug.php?id=70571 about LIKE and http://bugs.mysql.com/bug.php?id=70572  about FULLTEXT. If you need these features, please go and vote for them using "Affects Me!" Button.

Current behavior of JSON_SEARCH is tricky and Ulf, expectedly, did mistake when tried to use it. Correct syntax should be select json_search('{"key":"value"}', '"value"' ); and select json_search('{"a":{"b":"c"}}', '"c"' ) instead of select json_search('{"key":"value"}', "value" ); and select json_search('{"a":{"b":"c"}}', "c" ) which he used:

mysql> select json_search('{"key":"value"}', '"value"' )\G
*************************** 1. row ***************************
json_search('{"key":"value"}', '"value"' ): key::
1 row in set (0.00 sec)

mysql> select json_search('{"a":{"b":"c"}}', '"c"' );
+----------------------------------------+
| json_search('{"a":{"b":"c"}}', '"c"' ) |
+----------------------------------------+
| b:a::                                  |
+----------------------------------------+
1 row in set (0.00 sec)

This syntax is used to let you search for objects:

mysql> select json_search('{"a":{"b":"c"}}', '{"b":"c"}' );
+----------------------------------------------+
| json_search('{"a":{"b":"c"}}', '{"b":"c"}' ) |
+----------------------------------------------+
| a::                                          |
+----------------------------------------------+
1 row in set (0.00 sec)

You can also notice "::" in the end of the key path. This is the root element and done by design.

And last Ulf's concern is about syntactic sugar:"With a bit of syntactic sugar, say “SELECT document.member[offset]” this could look much more appealing. On the contrary, I am talking syntaxtic sugar only! Syntactic sugar is really hard to add with todays MySQL."

I so agree here! I wish I could implement the functions in such a way! But it is not possible to change MySQL syntax with UDFs!

And while we can not do much for syntactic sugar, you still can help to make better other features. Please comment and vote for the user manual format and location; parameter syntax; safe versions of the functions and extended search capabilites.

Thank you! :)

MySQL Connect 2013: ps_tools

$
0
0

During my talk and hands on labs at MySQL Connect 2013 I used several views, functions, and procedures from ps_helper and ps_tools.

The slides from the two talks can be downloaded from Thanks For Attending MySQL Connect or from the MySQL Connect 2013 Content Catalogue.

You can read more about ps_helper on Mark Leith’s blog and you can download ps_helper from github.

To get ps_tools, download ps_tools.tgz from here. Once unpacked there is a ps_tools_5x.sql for each of the versions supported. The tools presented at MySQL Connect were all based on MySQL 5.6 and 5.7. Note that several of the included tools are not particularly useful on their own but are more meant as utility functions for some of the other tools. The actual implementations are organised so they are in the subdirectory corresponding to the earliest version it applies to. For a few tools, such as ps_reset_setup(), there are multiple versions depending on the MySQL version it applies to.

Several of the tools have a help text at the top of the source file.

The main tools are:

  • Function ps_thread_id(): returns the thread id for a given connection. Specify NULL to get the thread id for the current connection. For example:
    mysql> SELECT ps_tools.ps_thread_id(8063);
    +-----------------------------+
    | ps_tools.ps_thread_id(8063) |
    +-----------------------------+
    |                        8113 |
    +-----------------------------+
    1 row in set (0.00 sec)
  •  View ps_setup_consumers: similar to performance_schema.setup_consumers, but includes a column to display whether the consumer is effectively enabled. See also slide 10 from CON 5282.
  • Function substr_count(): See also A Couple of Substring Functions: substr_count() and substr_by_delim().
  • Function substr_by_delim(): See also A Couple of Substring Functions: substr_count() and substr_by_delim().
  • Function ps_account_enabled(): Check whether a given account is enabled according to performance_schema.setup_actors. For example:
    mysql> SELECT ps_tools.ps_account_enabled('localhost', 'root');
    +--------------------------------------------------+
    | ps_tools.ps_account_enabled('localhost', 'root') |
    +--------------------------------------------------+
    | YES                                              |
    +--------------------------------------------------+
    1 row in set (0.03 sec)
  • View ps_accounts_enabled: Lists each account and whether it is enabled according to performance_schema.setup_actors.
  • Procedure ps_setup_tree_instruments(): Creates a tree displaying whether instruments are enabled. See Easier Overview of Current Performance Schema Setting for more details.
  • Procedure ps_setup_tree_actors_by_host(): Creates a tree displaying whether instruments are enabled. See Easier Overview of Current Performance Schema Setting for more details.
  • Procedure ps_setup_tree_actors_by_user(): Creates a tree displaying whether instruments are enabled. See Easier Overview of Current Performance Schema Setting for more details.
  • Procedure ps_setup_tree_consumers(): Creates a tree displaying whether instruments are enabled. See Easier Overview of Current Performance Schema Setting and slide 11 from CON 5282 for more details.
  • Procedure ps_dump_thread_stack(): This is the same as in ps_helper with one bug fix. See also slides 30-31 from CON 5282.
  • Procedure ps_enable_all(): Enable all consumers and instruments.
  • Procedure ps_reset_setup(): Reset consumers, instruments, actors, and threads to the defaults (without taking my.cnf into consideration).
  • View is_innodb_lock_waits: show information about waiting locks for InnoDB.
  • View is_engine_data: summary of the amount of data and indexes grouped by storage engine.
  • View ps_memory_by_thread_by_event_raw: the amount of memory used grouped by thread and event without any formatting and ordering.
  • View ps_memory_current_by_thread_by_event: The same as above but formatted output and ordered by current usage.
  • View ps_memory_high_by_thread_by_event: The same as above but formatted output and ordered by the high watermark.
  • Procedure schema_unused_routines(): Lists unused stored procedures and functions.

JSON UDFs have own bugs.mysql.com category

Late feedback

$
0
0


MySQL Community team asked me to write about Devconf 2013 few months ago. Conference was in June, 2013, but I remembered about this my promise only now: month later after my participating in MySQL Connect and Expert Troubleshooting seminar (change country to United Kingdom if you see blank page). I think it is too late for the feedback, but I still have few thoughts which I want to record.

DevConf (former PHPConf) always was a place where I tried new topics. At first, because I know audience there very well and they will be bored if I repeat a story which I was telling last year, but also because it is much easier to get feedback in your own native language. But last years my habit seems started to change and I presented improved version of my 2012 MySQL Connect talk about MySQL backups. Of course, I also had a seminar with unique topic, made for this conference first time: Troubleshooting MySQL Performance with EXPLAIN and Using Performance Schema to Troubleshoot MySQL. And these topics, improved, were presented at the expert seminar. It is interesting how my habit changes and one public speaking activity interferes next one.

What is good about DevConf is it forces you to create new ideas and do it really well, because audience is not forgiving at all, so they catch everything you miss or prepared not good enough. This can be bad if you want to make a marketing-style topic for free, but allows to present technical features in really good details: all these sudden discussions really help.

In year 2013 Oracle had a booth at the conference and was presented by a bunch of people. Dmitry Lenev presented topic "New features of replication in MySQL 5.6" and Victoria Reznichenko worked on the booth.




What was new at the conference this year is greater interest in
NoSQL, scale and fast development solutions. This, unfortunately, means
not so huge interest in MySQL as it was earlier. However, at the same
time, "Common" track was really MySQL track: not only Oracle, but people
from other companies presented about it.

To be safe or to be fast?

$
0
0

When I designed first version of JSON UDFs which was reviewed only internally, I let all functions to validate input and output JSON. But my colleagues told me to remove this functionality, because it makes such functions, as json_search, json_replace or json_contains_key deadly slow if they find the occurrence in the beginning of the long document. And first published version of JSON UDFs: 0.2.0 has not this functionality. What we expected is that users would call json_valid if they want to be 100% sure the document is valid.

But I was not surprised that some users expect JSON functions to work as it was in the first version: validate first, then process. For example, Ammon Sutherland writes: "json_set - according to the documentation a sort of 'INSERT... ON DUPLICATE KEY UPDATE' function which checks and parses the JSON (but not the whole document in this version for some reason)." Ulf Wendel also writes: "Taken from the README. For most functions manipulating or generating JSON documents is true: Warning! This version does not check whole document for validity. Hey, it is a labs.mysql.com pre-production release."

So this is certain what at least some of users want to have the behavior which was rejected. But since I believe others still can want better speed, I decided to put in my plans implementation of second set of the functions: safe JSON functions.

First, and more logical, candidate for this implementation is json_safe_merge, because currently json_merge is mostly not usable: if you checked already that documents are valid, you can easily split them. Therefore I created a separate feature request about this function: http://bugs.mysql.com/bug.php?id=70575

But I am not sure regarding other functions. This is why I created one more public feature request: "Implement SAFE versions for all JSON functions". Will I implement this or not would depend from your opinions. Therefore, if you want me to do so, please vote in the bug report by clicking "Affects Me!" button and comment if you want this feature will be implemented only for a few functions (or sooner for a particular function).


Install MySQL Enteprise Monitor (MEM) 3.0 Using Limited Resources

$
0
0

MySQL Enterprise Monitor (MEM) is the monitoring solution offered as part of MySQL Enterprise Edition and MySQL Cluster Carrier Grade Edition. In this blog, I will not go into details of the features of MEM, but rather focus on the case where you want to install MEM 3.0 to try it out on your personal desktop or laptop.

A trial version (the full product, but the can only be used for 30 days) is available from Oracle’s Software Delivery Cloud. If you are a MySQL customer, it is recommended that you download MEM from My Oracle Support (MOS).

Once you have downloaded and extracted the installation binaries, you can start the installation. You have the choice between using a GUI, text based, and unattended install. Here I will use the GUI install, but if you want to try one of the other install options, launch the installer with the –help option to get more information.

MEM 3.0 consists of two parts:

  • The Service Manager
  • Agents

Only the Service Manager is required, so that will be the first thing to install. As this is intended to show how you can test MEM, I will use a normal user rather than root for the installation.

It is also recommended to take a look at the MEM 3.0 Reference Manual.

Installation Wizard

When launching the installer the first screen asks which language to use – you have the choice of English and Japanese:

Installing the MEM 3.0 Service Manager - Step 1: Choose language

Step 1: Choose language

Next is an information screen that you should ensure you keep track of the usernames and passwords entered during the installation process:

Installing the MEM 3.0 Service Manager - Step 2: Remember the usernames and passwords entered during the installation process

Step 2: Remember the usernames and passwords entered during the installation process

The actual install process is now ready to start:

Installing the MEM 3.0 Service Manager - Step 3: Ready to start the actual install process

Step 3: Ready to start the actual install process

The following steps are to configure the installation – the first of which is to decide where to install the Service Manager:

Installing the MEM 3.0 Service Manager - Step 4: Choose the installation directory

Step 4: Choose the installation directory

The Service Manager will need three ports:

  • Tomcat Server Port: For the web UI when using non-SSL connections
  • Tomcat Shutdown Port: Used internally to shutdown the web server
  • Tomcat SSL Port: For the web UI when using SSL connections
Installing the MEM 3.0 Service Manager - Step 5: Choose the port numbers

Step 5: Choose the port numbers

The Service Manager uses a MySQL database to store the data collected. The next screen allows you to choose between using the MySQL database bundled with MEM or an existing one. Using the bundled instance has the advantage that MEM will configure it automatically and upgrades can be easier, however it will mean running two MySQL instances if you already have MySQL installed. For a test instance using the bundled instance also has the advantage that it’s easy to uninstall the whole installation again, so we will use the bundled instance in this example:

Installing the MEM 3.0 Service Manager - Step 6: Choose whether to use the bundled MySQL database or an existing one

Step 6: Choose whether to use the bundled MySQL database or an existing one

The final of the pre-installation configuration is to choose the username and password to use for the connection to the MySQL database. This is the username and password that you were reminded of earlier to make sure you remember:

Installing the MEM 3.0 Service Manager - Step 7: Specify username and password for the Service Manager to connect to the MySQL database storing the data collected through the monitoring

Step 7: Specify username and password for the Service Manager to connect to the MySQL database storing the data collected through the monitoring

Next a note that because we have chosen to install the Service Manager as a non-root user, it will not be started automatically when the server is restarted:

Installing the MEM 3.0 Service Manager - Step 8: Info that MEM will not start automatically when not installed as root

Step 8: Info that MEM will not start automatically when not installed as root

Ready to Install:

Installing the MEM 3.0 Service Manager - Step 9: Configuration completed

Step 9: Configuration completed – ready to install

The Service Manager is now being installed – this will take a little while as it both includes copying all the files in place as well as configuring and starting the web server and the bundled MySQL database:

Installing the MEM 3.0 Service Manager - Step 10: Installation is in progress

Step 10: Installation is in progress

MEM includes an uninstall script if you need to remove the Service Manager again:

Installing the MEM 3.0 Service Manager - Step 11: Files are now installed

Step 11: Files are now installed

To improve security MEM 3.0 by default uses SSL. The installation process adds a self-signed certificate, but you can choose to install your own certificate later:

Installing the MEM 3.0 Service Manager - Step 12: Information that MEM is using SSL with a self-signed certificate by default

Step 12: Information that MEM is using SSL with a self-signed certificate by default

The installer is now done and it is time to launch the UI:

Installing the MEM 3.0 Service Manager - Step 13: Ready to launch the Service Manager

Step 13: Ready to launch the Service Manager

Unattended Install

If you go through the installation process often, it is more convenient to use the unattended installation as it allows you to automate the installation. To perform the same installation as above using the unattended installation you can execute:

shell$ ./mysqlmonitor-3.0.3.2912-linux-x86_64-installer.bin \
> --mode unattended --unattendedmodeui none \
> --installdir /home/jkrogh/mysql/enterprise/monitor \
> --adminuser service_manager --adminpassword mypassword \
> --dbport 13306 --mysql_installation_type bundled \
> --tomcatport 18080 --tomcatshutdownport 18005 --tomcatsslport 18443

As several of the values are the default ones, you can skip some of the options, but they are included here for completeness.

When the above command completes, continue with the post-installation configuration as you would have done using the installation wizard.

Post-Installation Configuration

Once the Service Manager UI has opened in your browser there is a little post-installation configuration to take care of:

  • You need to create an admin user for the web UI
  • You need to create a user agents can use if you choose to install any agents
  • You can choose whether to have the Service Manager automatically check for updates and how for how long time to keep historical data
Installing the MEM 3.0 Service Manager - Step 14: Post installation configuration

Step 14: Post installation configuration

The last step if to choose your timezone and which locale to use in the user interface:

Installing the MEM 3.0 Service Manager - Step 15: Choose timezone and the locale

Step 15: Choose timezone and the locale

After this you are ready to play around with the monitor. If you choose to install agents, the steps are similar.

Reducing Memory Footprint

One thing to be aware of though is that the default configuration used by the Service Manager is aimed at a small to medium production installation. If you for example installed MEM in a virtual machine or computer with limited memory available, the default configuration may not be optimal.

With the installation above just completed, the memory usage of the web server (Tomcat) and the bundled MySQL database is:

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
15653 jkrogh    20   0 3794m 924m  13m S  1.0 23.4   4:46.01 java
15586 jkrogh    20   0 3425m 608m 9588 S  2.6 15.4   2:23.88 mysqld

So around 1.5G resident memory. That is without adding any additional agents and/or MySQL instances to monitor.

So when I use MEM for testing, the first thing I do is to change a few configuration values to reduce the memory usage. The configuration options I change are located in two files (paths are given relative to the installation directory):

  • mysql/my.cnf – options related to the bundled MySQL database
  • apache-tomcat/bin/setenv.sh – options related to the web server

mysql/my.cnf

For the MySQL database I change two setting:

  • innodb_buffer_pool_size – this is by default 768M, but if you only monitor a couple of MySQL instances, something like 100M is enough
  • table_definition_cache – this is by default 2048. One side effect of the value being so large is that the auto-sizing of the Performance Schema considers the installation as a large instance, so the Performance Schema tables are made large. Reducing the value to 800 is enough for testing and will reduce the memory usage of the Performance Schema with several hundred megabytes.

So my mysql/my.cnf looks something like this after the changes:

...
[mysqld]
...
# Main Caches
table_definition_cache = 800
...

# InnoDB
...
innodb_buffer_pool_size = 100M
...

 apache-tomcat/bin/setenv.sh

It is only necessary to edit one line in the Tomcat configuration file – the default settings are:

JAVA_OPTS="-Xmx768M -Xms768M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/jkrogh/mysql/enterprise/monitor/apache-tomcat/temp -XX:+UseParallelOldGC -XX:MaxPermSize=512M"

I change that to:

JAVA_OPTS="-Xmx256M -Xms256M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/jkrogh/mysql/enterprise/monitor/apache-tomcat/temp -XX:+UseParallelOldGC -XX:MaxPermSize=200M"

I.e. the three setting I have changed are:

  • -Xmx (maximum  heap size) from 768M to 256M
  • -Xms (minimum heap size) from 768M to 256M
  • -XX:MaxPermSize from 512M to 200M

Enabling the New Configurations

It requires restarting the Service Manager to make the new configurations take effect. You can do the restart by going into the installation directory and execute:

monitor$ ./mysqlmonitorctl.sh restart
Using CATALINA_BASE:   /home/jkrogh/mysql/enterprise/monitor/apache-tomcat
Using CATALINA_HOME:   /home/jkrogh/mysql/enterprise/monitor/apache-tomcat
Using CATALINA_TMPDIR: /home/jkrogh/mysql/enterprise/monitor/apache-tomcat/temp
Using JRE_HOME:        /home/jkrogh/mysql/enterprise/monitor/java
Using CLASSPATH:       /home/jkrogh/mysql/enterprise/monitor/apache-tomcat/bin/bootstrap.jar:/home/jkrogh/mysql/enterprise/monitor/apache-tomcat/bin/tomcat-juli.jar
Using CATALINA_PID:    /home/jkrogh/mysql/enterprise/monitor/apache-tomcat/temp/catalina.pid
Stopping tomcat service . [ OK ]
Stopping mysql service . [ OK ]
./mysqlmonitorctl.sh : mysql  started
131128 13:58:59 mysqld_safe Logging to '/home/jkrogh/mysql/enterprise/monitor/mysql/runtime/mysqld.log'.
131128 13:58:59 mysqld_safe Starting mysqld daemon with databases from /home/jkrogh/mysql/enterprise/monitor/mysql/data/
Using CATALINA_BASE:   /home/jkrogh/mysql/enterprise/monitor/apache-tomcat
Using CATALINA_HOME:   /home/jkrogh/mysql/enterprise/monitor/apache-tomcat
Using CATALINA_TMPDIR: /home/jkrogh/mysql/enterprise/monitor/apache-tomcat/temp
Using JRE_HOME:        /home/jkrogh/mysql/enterprise/monitor/java
Using CLASSPATH:       /home/jkrogh/mysql/enterprise/monitor/apache-tomcat/bin/bootstrap.jar:/home/jkrogh/mysql/enterprise/monitor/apache-tomcat/bin/tomcat-juli.jar
Using CATALINA_PID:    /home/jkrogh/mysql/enterprise/monitor/apache-tomcat/temp/catalina.pid

The memory usage is now somewhat smaller:

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
19521 jkrogh    20   0 2847m 544m  15m S  2.0 13.8   1:51.61 java
19484 jkrogh    20   0 2407m 160m 9296 S  0.3  4.1   0:09.21 mysqld

 

JSON UDF functions version 0.2.1 have been released.

$
0
0

Today new version of JSON UDF functions: 0.2.1 was released. This is maintenance release which added no new functionality and only contains bug fixes. However, it also includes improvements for build ans test procedures. As usual, you can download source and binary packages at MySQL Labs. Binary packages were build for MySQL server 5.6.14. If you want to use other version of the server, you need to recompile functions.

What was changed? Let me quote the ChangeLog.

Functionality added or changed:

Added cmake option WITH_PCRE which alolows to specify if existent or bundled version of PCRE  should be used. Bundled is default on Windows. To compile with bundled version, run: "cmake . -DMYSQL_DIR=/path/to/mysql/dir -DWITH_PCRE=bundled", to turn bundled version off on Windows, run: "cmake . -DMYSQL_DIR=/path/to/mysql/dir -DWITH_PCRE=system"


This means you don't need to type additional commands and move files from one directory to another to build JSON UDFs on Windows.

Added possibility to run tests, using command make test. By default tests are running without valgrind. If you need to run tests under valgrind, add option VALGRIND=1 to cmake command: "cmake . -DMYSQL_DIR=/path/to/mysql/dir -DVALGRIND=1"


You can automatically test on both UNIX and Windows. To test on UNIX simply run make test. On Windows: open solution my_json_udf.sln in Visual Studio, then choose target ALL_TESTS and run it. If you know how to run custom project of a solution on command line please let me know too =)

Added cmake target build_source_dist which creates source tarball. Usage: "make build_source_dist".

This addition is mostly for developers and needed to create packages for MySQL Labs.

Minimum supported cmake version changed to 2.8. It is necessary to run tests and build PCRE.

Bugs Fixed:

70392/17491678 MySQL JSON UDFs binary is called libmy_json_udf.so but DDL uses libmy_json.so

70573/17583505 Typo in README for JSON_MERGE

70605/17596660 Include all references to documentaiton to README file

70569/17583335 JSON_VALID allows mixed case in the keyword names

70568/17583310 JSON_VALID treats invalid values as valid

70570/17583367 JSON_VALID allows \u two-hex-digit while standard allows only \u four-hex-digit

70574/17583525 JSON_MERGE treats document without opening bracket as valid

70486/17548362 When using json_replace(), '}' of the end disappear.

70839/17751860 JSON_VALID allows to have two elements with the same key


I hope this version will be more stable than the first one and I hope you enjoy it.

Last element for JSON array: what do you think?

$
0
0

After I released maintenance release of JSON UDFs last week it is time to think about which features I should implement in upcoming major version.

Many users asked me about the possibility to explicitly specify if they want to append last element to a JSON array. This feature can be made for two functions: json_append and json_set.

I have four ideas of how to implement this. All have pros and contras.



  1. Create new function called json_append_last which will work exactly like json_append, but it will add the element to the end of array. I.e., for JSON document {"colors": ["red", "green", "blue"]} json_append_last(doc, 'colors', 'orange') returns {"colors": ["red", "green", "blue", "orange"]}


    1. Pros:


      1. Such a code is easy to understand


    2. Contras:


      1.  There is no sense to implement analogous function json_set_last, therefore this possibility is for json_append only

      2. You should explicitly branch your code if you want to add an element to known position of the array and if you want to add last element



  2. Use string keyword. For example, 'LAST'. I.e., for JSON document {"colors": ["red", "green", "blue"]} json_append(doc, 'colors', 'LAST', 'orange') returns {"colors": ["red", "green", "blue", "orange"]} 


    1. Pros:


      1. It is easy to implement same thing for json_set

      2. You should not branch code if you need both to add an element to known position and to the end of array


    2. Contras:


      1. It is easy to misuse. For example, if you mistakenly passed JSON document {"colors": {"LAST": "transparent"}} to json_append(doc, 'colors', 'LAST', 'orange') it returns {"colors": {"LAST": "orange"}} instead of error



  3. Use NULL instead of string keyword. I.e., for JSON document {"colors": ["red", "green", "blue"]} json_append(doc, 'colors', NULL, 'orange') returns {"colors": ["red", "green", "blue", "orange"]} 


    1. Pros:


      1. It is easy to implement same thing for json_set

      2. You should not branch code if you need both to add an element to known position and to the end of array

      3. No way to misuse: JSON functions don't take NULL as an argument anywhere else.


    2. Contras:


      1. Looks like a hack: NULL is not the type of last element

      2. It is hard to guess what this function is doing without looking into user manual



  4. Use negative index. So, -1 means last element, -2 last before last and so on. I.e., for JSON document {"colors": ["red", "green", "blue"]} json_append(doc, 'colors', -1, 'orange') returns {"colors": ["red", "green", "blue", "orange"]} and json_set(doc, 'colors', -2, 'orange') returns {"colors": ["red", "green", "orange"]}. json_append returns original document in case of -2.


    1. Pros:


      1. Works for both json_append and json_set

      2. You should not branch code if you need both to add an element to known position and to the end of array

      3. No way to misuse: JSON functions don't take negative number as an argument anywhere else.


    2. Contras:


      1. Confusing: while users of programming languages which use negative array indexes can guess what these functions do it is still needed to look in the manual to find out how -1, -2, etc. are interpreted

      2. Error-prone: it is easy to mix up such numbers





As a conclusion I can say that I think solutions 1 and 3 are less error-prone and solution 4 introduces useful side-effects. I think that it is better to implement either 1 or 3, but certainly not both.

But what do you think about it? Please either comment here or at bugs.mysql.com

New cmake options for JSON UDFs.

$
0
0

Two weeks ago I announced new maintenance release of JSON UDFs: 0.2.1. It not only contains bug fixes, how you can expect from a maintenance release, but also contains improvements in build and test scripts.

First improvement is the easier way to build JSON UDFs on Windows. In the first version building on Windows was a pane: you had to build PCRE library, copy four files to directories where Visual Studio can find them and only then build JSON functions themselves. Now you can build this way too, but only if you really wish.

By default cmake will create appropriate makefiles for bundled PCRE sources and for JSON functions. Only command you need to prepare sources is:

"C:\Program Files (x86)\CMake 2.8\bin\cmake.exe" -G "Visual Studio 11 Win64" . -DMYSQL_DIR="/path/to/mysql/installation"   

And then you can build functions:

devenv my_json_udf.sln /build Release

This is done by introducing new cmake option WITH_PCRE which can take two parameters: system and bundled. For Windows default is bundled which means bundled version of PCRE will be used. Default parameter for UNIX is system, because you usually have regular expression library installed and no need to build it statically into JSON functions. But you can overwrite defaults on both systems, specifying option -DWITH_PCRE=system|bundled for the cmake command.

Another improvement is test automation. In the first available version you had to use strange file Makefile.unix and weird instructions to run tests. In 0.2.1 this file is not needed anymore: simply run `make test` on UNIX or open solution my_json_udf.sln in Visual Studio and execute target ALL_TESTS. Here I have a question for mature Windows users: do you know how to run this on command line? But regardless to command line or not this also means you don't need Cygwin to run tests on Windows. Perl, however, is still required and must be in the path, because it is required by the MySQL Test Framework.

Of course, MySQL server directory should contain binaries. In other words: if you use source dir when build JSON functions you need to compile MySQL server to be able to run tests.

On Linux, as well on other OSes which can run Valgrind, you can run tests with valgrind too. Instructions are a bit weird: you need to specify option -DVALGRIND=1 when build functions to run tests under valgrind. Command, which you need to use to run tests under valgrind is still the same: make test

And, finally, last improvement is possibility to create a source package by running command make build_source_dist. This option can look not very useful from the user point of view, but it really helps producing packages for MySQL Labs out of internal repository with sources. If you run make build_source_dist file, named mysql-json-udfs-0.2.1-labs-json-udfs-src.tar.gz, which contains all source code, will be produced. On Windows, again, you need to open my_json_udf.sln solution and build this target.

JSON UDFs: is it hard to type keys?

$
0
0

Currently, if you want to search a key using JSON UDFs you have to specify each its part as a separate argument to the function: JSON_CONTAINS_KEY(doc, 'root', 'child', 'child of child', 'child of child of child', etc.....). This way of working with parameters is easy for developer, less error-prone, but can be not very beautiful.

I was suggested by some of users to change it to '/parent/child/child-of-child/...' or to 'parent:child:child-of-child:...' There are, probably, can be other suggestions. What do you like the best? How do you feel about current style with separate argument for each key element? Should we change or extend this syntax?

A small optimizer change worth noticing

$
0
0

MySQL uses internal temporary tables to execute some queries. Usually the tables are stored in memory and on disk if some conditions are met :

Some conditions prevent the use of an in-memory temporary table, in which case the server uses an on-disk table instead:

  • Presence of a BLOB or TEXT column in the table

  • Presence of any string column in a GROUP BY or DISTINCT clause larger than 512 bytes

  • Presence of any string column with a maximum length larger than 512 (bytes for binary strings, characters for nonbinary strings) in the SELECT list, if UNION or UNION ALL is used

http://dev.mysql.com/doc/refman/5.6/en/internal-temporary-tables.html

This was true until MySQL 5.6.15. A voluntary side effect of this bug fix :

Bug 17566396 – MATERIALIZATION IS NOT CHOSEN FOR LONG UTF8 VARCHAR JOINS

was to loosen the condition on the second item. You can now use up to varchar(512) whatever the character set (utf8, utf8mb4) in group by / distinct and of course in joins. I often see varchar(255), usually in latin1 and more and more in utf8.

Here is an example :

USE test;
DROP TABLE IF EXISTS t;
CREATE TABLE t(txt varchar(255)) ENGINE=InnoDB CHARSET = utf8;
INSERT INTO t(txt) VALUES (repeat('0', 255));
FLUSH STATUS;
SHOW SESSION STATUS LIKE '%tmp%';
SELECT DISTINCT txt FROM t;
SHOW SESSION STATUS LIKE '%tmp%';

Result in 5.6.14 :

mysql> show session status like '%tmp%';
+-------------------------+-------+
| Variable_name           | Value |
+-------------------------+-------+
| Created_tmp_disk_tables | 1     |   <==== goes to disk ...
| Created_tmp_files       | 0     |
| Created_tmp_tables      | 1     | 
+-------------------------+-------+
3 rows in set (0.00 sec)

Result in 5.6.15 :

mysql> show session status like '%tmp%';
+-------------------------+-------+
| Variable_name           | Value |
+-------------------------+-------+
| Created_tmp_disk_tables | 0     | <=== goes to memory !
| Created_tmp_files       | 0     |
| Created_tmp_tables      | 1     |
+-------------------------+-------+
3 rows in set (0.00 sec)

This is the internal bug for the documentation change request :

Bug 17935006 – INTERNAL TEMPORARY TABLE LIMITATIONS CHANGED IN 5.6.15

A big thank to the Optimizer team for fixing this very ancient restriction !

JSON UDF functions version 0.3.0 have been released.

$
0
0

Today new version of JSON UDF functions: 0.3.0 was released. This is major release which contains new functionality. You can download functions from the MySQL Labs website.

What was added?

Functions JSON_EXTRACT, JSON_CONTAINS_KEY, JSON_APPEND, JSON_REMOVE, JSON_REPLACE, JSON_SET now accept both string and integer arguments as array indexes. (Bug #70393/17491709)

Now syntax JSON_EXTRACT(doc, 'key', 1); is accepted.

New function JSON_SAFE_MERGE checks JSON documents for validity before merging them. In addition JSON_MERGE now merges valid documents, having array as root element. (Bug #70575/17583568)

JSON_VALID now accepts array as a root element of the JSON document (Bug #70567/17583282)

Also this release contains following bug fixes:

71050/17882710 json_extract returning same column twice if key names are not fully distinct. (Fixed for JSON_EXTRACT, JSON_CONTAINS_KEY, JSON_APPEND, JSON_REMOVE, JSON_REPLACE, JSON_SET)

71180/17975709 JSON_CONTAINS_KEY founds not existent key in the array element

70579/17584678 Add ability to append last element to array in JSON_APPEND

And, of course, this release contains all changes which version 0.2.1 has


When one of our most talented MySQL Support Engineers asks for help, let’s give him and his wife a hand!

$
0
0

Dear MySQL Family,

Soon before Christmas I read these lines Shawn Green (one of our most talented Support Engineer) had written about his wife at Amie’s lottery chances were better..:

What can you do when you are one of a 1000 people out of the 7.5 billion on the planet who have been diagnosed with this particular disease. Imagine having your guts knotted up as though you had Crohn’s disease, add to that fever and aches as though you had flu, add to this the feeling that you have the arthritic joints of a 80 year old and you are 42… Now you have a idea of what T.R.A.P.S syndrome is like. Now imagine having cancer as well. The money that is being raised is going to offset the cost of Mayo clinic in Minnesota…as well as current medical bills. We have discovered that her cancer has spread..but still treatable. We are home finally and will find out when next surgery will be.

There are two technical parts on the page:

  1. Medical
  2. How to make a donation using a credit card.

I have to admit I’m no good on medicine or medical terms. However, the second technical part using my credit card was easier for me. If you, just like me, prefer to make anonymous donations there is a checkbox for it.

So when one of our most talented MySQL Support Engineers asks for help, let’s give him and his wife a hand!

Wish you all a happy new 2014.

Sincerely, Gustaf

Who Holds the Metadata Lock? MySQL 5.7.3 Brings Help

$
0
0

In MySQL 5.5 metadata locks were introduced to ensure consistency for some operations that previously could cause problems. For example if a transaction SELECTed from a table and another connection changed the structure of the table before the transaction was committed; in that case the two operations would be written in reverse order to the binary log which could prevent replaying the binary log.

However there was no way to see who held metadata locks. SHOW PROCESSLIST would show who were waiting for metadata locks, and often you could guess who held it, but it wasn’t so always. Consider for example the following:

mysql> SELECT Id, db, Command, Time, State, Info FROM information_schema.PROCESSLIST;
+------+--------------------+---------+------+---------------------------------+-------------------------------------------------------------------------------+
| Id   | db                 | Command | Time | State                           | Info                                                                          |
+------+--------------------+---------+------+---------------------------------+-------------------------------------------------------------------------------+
|    1 | NULL               | Daemon  |    8 | Waiting for next activation     | NULL                                                                          |
|    2 | performance_schema | Query   |    0 | executing                       | SELECT Id, db, Command, Time, State, Info FROM information_schema.PROCESSLIST |
|    3 | world              | Sleep   |   39 |                                 | NULL                                                                          |
|    4 | world              | Query   |   35 | Waiting for table metadata lock | ALTER TABLE City COMMENT='Cities'                                             |
| 1111 | world              | Sleep   |   32 |                                 | NULL                                                                          |
| 1108 | world              | Sleep   |   47 |                                 | NULL                                                                          |
| 1113 | world              | Sleep   |   18 |                                 | NULL                                                                          |
| 1112 | world              | Connect |   23 | Waiting for table metadata lock | SELECT COUNT(*) INTO @count FROM view_city WHERE CountryCode = in_CountryCode |
| 1109 | world              | Sleep   |   44 |                                 | NULL                                                                          |
| 1114 | world              | Connect |    8 | Waiting for table metadata lock | SELECT COUNT(*) INTO @count FROM view_city WHERE CountryCode = in_CountryCode |
+------+--------------------+---------+------+---------------------------------+-------------------------------------------------------------------------------+
10 rows in set (0.00 sec)

In this case it happens to be the process with Id = 3 that holds the metadata lock, but that is not obvious from the above output.

In MySQL 5.7.3 this has changed. There is a new Performance Schema table metadata_locks which will tell which metadata locks are held and which are pending. This can be used to answer the question for a case like the one above.

First instrumentation of metadata locks should be enabled. To do that you must enable the wait/lock/metadata/sql/mdl instrument in setup_instruments. Additionally the global_instrumentation consumer must be enabled in setup_consumers.

Currently the wait/lock/metadata/sql/mdl instrument is not enabled by default. I have created a feature request to consider enabling it by default, but obviously whether that will happen also depends on the performance impact.

To enable metadata lock instrumentation:

mysql> UPDATE setup_consumers SET ENABLED = 'YES' WHERE NAME = 'global_instrumentation';
Query OK, 1 row affected (0.01 sec)
Rows matched: 1  Changed: 1  Warnings: 0

mysql> UPDATE setup_instruments SET ENABLED = 'YES' WHERE NAME = 'wait/lock/metadata/sql/mdl';
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

Note: The global_instrumentation consumer is enabled by default.

With the metadata lock instrumentation enabled, it is now easy to answer who holds the metadata lock (I’ve excluded the connections own metalocks here as I’m only interested in the metadata lock contention going on between other queries):

mysql> SELECT OBJECT_TYPE, OBJECT_SCHEMA, OBJECT_NAME, LOCK_TYPE, LOCK_STATUS,
    ->        THREAD_ID, PROCESSLIST_ID, PROCESSLIST_INFO
    ->   FROM performance_schema.metadata_locks
    ->        INNER JOIN performance_schema.threads ON THREAD_ID = OWNER_THREAD_ID
    ->  WHERE PROCESSLIST_ID <> CONNECTION_ID();
+-------------+---------------+-------------+---------------------+-------------+-----------+----------------+-------------------------------------------------------------------------------+
| OBJECT_TYPE | OBJECT_SCHEMA | OBJECT_NAME | LOCK_TYPE           | LOCK_STATUS | THREAD_ID | PROCESSLIST_ID | PROCESSLIST_INFO                                                              |
+-------------+---------------+-------------+---------------------+-------------+-----------+----------------+-------------------------------------------------------------------------------+
| TABLE       | world         | City        | SHARED_READ         | GRANTED     |        22 |              3 | NULL                                                                          |
| GLOBAL      | NULL          | NULL        | INTENTION_EXCLUSIVE | GRANTED     |        23 |              4 | ALTER TABLE City COMMENT='Cities'                                             |
| SCHEMA      | world         | NULL        | INTENTION_EXCLUSIVE | GRANTED     |        23 |              4 | ALTER TABLE City COMMENT='Cities'                                             |
| TABLE       | world         | City        | SHARED_UPGRADABLE   | GRANTED     |        23 |              4 | ALTER TABLE City COMMENT='Cities'                                             |
| TABLE       | world         | City        | EXCLUSIVE           | PENDING     |        23 |              4 | ALTER TABLE City COMMENT='Cities'                                             |
| TABLE       | world         | view_city   | SHARED_READ         | GRANTED     |      1185 |           1166 | SELECT COUNT(*) INTO @count FROM view_city WHERE CountryCode = in_CountryCode |
| TABLE       | world         | City        | SHARED_READ         | PENDING     |      1185 |           1166 | SELECT COUNT(*) INTO @count FROM view_city WHERE CountryCode = in_CountryCode |
| TABLE       | world         | view_city   | SHARED_READ         | GRANTED     |      1186 |           1167 | SELECT COUNT(*) INTO @count FROM view_city WHERE CountryCode = in_CountryCode |
| TABLE       | world         | City        | SHARED_READ         | PENDING     |      1186 |           1167 | SELECT COUNT(*) INTO @count FROM view_city WHERE CountryCode = in_CountryCode |
| TABLE       | world         | view_city   | SHARED_READ         | GRANTED     |      1187 |           1168 | SELECT COUNT(*) INTO @count FROM view_city WHERE CountryCode = in_CountryCode |
| TABLE       | world         | City        | SHARED_READ         | PENDING     |      1187 |           1168 | SELECT COUNT(*) INTO @count FROM view_city WHERE CountryCode = in_CountryCode |
| TABLE       | world         | view_city   | SHARED_READ         | GRANTED     |      1188 |           1169 | SELECT COUNT(*) INTO @count FROM view_city WHERE CountryCode = in_CountryCode |
| TABLE       | world         | City        | SHARED_READ         | PENDING     |      1188 |           1169 | SELECT COUNT(*) INTO @count FROM view_city WHERE CountryCode = in_CountryCode |
| TABLE       | world         | view_city   | SHARED_READ         | GRANTED     |      1189 |           1170 | SELECT COUNT(*) INTO @count FROM view_city WHERE CountryCode = in_CountryCode |
| TABLE       | world         | City        | SHARED_READ         | PENDING     |      1189 |           1170 | SELECT COUNT(*) INTO @count FROM view_city WHERE CountryCode = in_CountryCode |
| TABLE       | world         | view_city   | SHARED_READ         | GRANTED     |      1190 |           1171 | SELECT COUNT(*) INTO @count FROM view_city WHERE CountryCode = in_CountryCode |
| TABLE       | world         | City        | SHARED_READ         | PENDING     |      1190 |           1171 | SELECT COUNT(*) INTO @count FROM view_city WHERE CountryCode = in_CountryCode |
| TABLE       | world         | view_city   | SHARED_READ         | GRANTED     |      1191 |           1172 | SELECT COUNT(*) INTO @count FROM view_city WHERE CountryCode = in_CountryCode |
| TABLE       | world         | City        | SHARED_READ         | PENDING     |      1191 |           1172 | SELECT COUNT(*) INTO @count FROM view_city WHERE CountryCode = in_CountryCode |
+-------------+---------------+-------------+---------------------+-------------+-----------+----------------+-------------------------------------------------------------------------------+
19 rows in set (0.00 sec)

So in this case there is a metadata lock GRANTED to process list id 3 whereas the other connections have a PENDING lock status for the metadata lock for the City table.

MySQL and PostgreSQL JSON functions: do they differ much?

$
0
0
As author of MySQL JSON functions I am also interested in how development goes in another parties. JSON functions and operators in PostgreSQL, indeed, have great features. Some of them, such as operators, I can not do using UDF functions only. But lets see how these functions are interchangeable.

Note: all PostgreSQL examples were taken from PostgreSQL documentation.

First topic is syntax sugar.

 PostgreSQL
MySQL

Operator ->

postgres=# select '[1,2,3]'::json->2;
 ?column?
----------
 3
(1 row)

postgres=# select '{"a":1,"b":2}'::json->'b';
 ?column?
----------
 2
(1 row)


JSON_EXTRACT

mysql> select json_extract('[1,2,3]', 2);
+----------------------------+
| json_extract('[1,2,3]', 2) |
+----------------------------+
| 3                          |
+----------------------------+
1 row in set (0.00 sec)

mysql> select json_extract('{"a":1,"b":2}', 'b');
+------------------------------------+
| json_extract('{"a":1,"b":2}', 'b') |
+------------------------------------+
| 2                                  |
+------------------------------------+
1 row in set (0.00 sec)

Operator ->>

postgres=# select '[1,2,3]'::json->>2;
 ?column?
----------
 3
(1 row)

postgres=# select '{"a":1,"b":2}'::json->>'b';
 ?column?
----------
 2
(1 row)


JSON_EXTRACT

mysql> select json_extract('[1,2,3]', 2);
+----------------------------+
| json_extract('[1,2,3]', 2) |
+----------------------------+
| 3                          |
+----------------------------+
1 row in set (0.00 sec)

mysql> select json_extract('{"a":1,"b":2}', 'b');
+------------------------------------+
| json_extract('{"a":1,"b":2}', 'b') |
+------------------------------------+
| 2                                  |
+------------------------------------+
1 row in set (0.00 sec)

Operator #>

postgres=# select '{"a":[1,2,3],"b":[4,5,6]}'::json#>'{a,2}';
 ?column?
----------
 3
(1 row)


JSON_EXTRACT

mysql> select json_extract('{"a":[1,2,3],"b":[4,5,6]}', 'a', 2);
+---------------------------------------------------+
| json_extract('{"a":[1,2,3],"b":[4,5,6]}', 'a', 2) |
+---------------------------------------------------+
| 3                                                 |
+---------------------------------------------------+
1 row in set (0.00 sec)

Operator #>>

postgres=# select '{"a":[1,2,3],"b":[4,5,6]}'::json#>>'{a,2}';
 ?column?
----------
 3
(1 row)

JSON_EXTRACT

mysql> select json_extract('{"a":[1,2,3],"b":[4,5,6]}', 'a', 2);
+---------------------------------------------------+
| json_extract('{"a":[1,2,3],"b":[4,5,6]}', 'a', 2) |
+---------------------------------------------------+
| 3                                                 |
+---------------------------------------------------+
1 row in set (0.00 sec)


Then PostgreSQL JSON functions

 PostgreSQL
 MySQL
array_to_json(anyarray [, pretty_bool])  Not supported
row_to_json(record [, pretty_bool])  Not supported
to_json(anyelement)  Not supported
json_array_length(json)  Not supported, planned as bug #70580
json_each(json)  Not supported, cannot be implemented using UDFs only
json_each_text(from_json json)  Not supported, cannot be implemented using UDFs only

json_extract_path(from_json json, VARIADIC path_elems text[])


postgres=# select json_extract_path('{"f2":{"f3":1},"f4":{"f5":99,"f6":"foo"}}','f4');
  json_extract_path   
----------------------
 {"f5":99,"f6":"foo"}
(1 row)

JSON_EXTRACT

mysql> select json_extract('{"f2":{"f3":1},\c

"f4":{"f5":99,"f6":"foo"}}','f4');
+---------------------------------+
| json_extract('{"f2":{"f3":1},

"f4":{"f5":99,"f6":"foo"}}','f4') |
+---------------------------------+
| {"f5":99,"f6":"foo"}            |
+---------------------------------+
1 row in set (0.00 sec)

json_extract_path_text(from_json json, VARIADIC path_elems text[])


postgres=# select json_extract_path_text('{"f2":{"f3":1},"f4":{"f5":99,"f6":"foo"}}','f4', 'f6');
 json_extract_path_text
------------------------
 foo
(1 row)

 JSON_EXTRACT

select json_extract('{"f2":{"f3":1},\c

"f4":{"f5":99,"f6":"foo"}}','f4', 'f6');
+---------------------------------------+
| json_extract('{"f2":{"f3":1},

"f4":{"f5":99,"f6":"foo"}}','f4', 'f6') |
+---------------------------------------+
| foo                                   |
+---------------------------------------+
1 row in set (0.01 sec)

json_object_keys(json)  Not supported, cannot be implemented using UDFs only
json_populate_record(base anyelement, from_json json, [, use_json_as_text bool=false]  Not supported, cannot be implemented using UDFs only
json_populate_recordset(base anyelement, from_json json, [, use_json_as_text bool=false]  Not supported, cannot be implemented using UDFs only
json_array_elements(json)  Not supported, cannot be implemented using UDFs only

And, finally, MySQL functions

 MySQL
 PostgreSQL
json_append  Not supported, but user can convert JSON value into various formats, then create new JSON document from it
json_contains_key  Not supported, however has function json_object_keys(json)
json_extract  Supported in numerous ways, see above
json_merge   Not supported, but user can convert JSON value into various formats, then create new JSON document from it
json_remove   Not supported, but user can convert JSON value into various formats, then create new JSON document from it
 json_replace
  Not supported, but user can convert JSON value into various formats, then create new JSON document from it
json_search  Not supported
json_set   Not supported, but user can convert JSON value into various formats, then create new JSON document from it
json_test_parser   Not supported
json_valid  Not supported, however it has JSON type and checks every element for validity


As a conclusion I can say that MySQL and PostgreSQL implementations mostly intersect in extracting elements while solve element validation and manipulation tasks in different ways.

JSON UDF functions version 0.2.2 have been released.

$
0
0
New version of JSON UDF functions 0.2.2 have been just released. It is last maintenance release of 0.2 series. However it contains two new features:

JSON_VALID now accepts array as a root element of the JSON document (Bug#70567/17583282)

JSON functions can now be installed and uninstalled using a script (Bug#71263/18022788 Easy installation sql script).

This feature is Community contribution. Thank you, Daniel van Eeden!

This release also contains following bug fixes:

71050/17882710 json_extract returning same column twice if key names are not fully distinct. (Fixed for JSON_EXTRACT, JSON_CONTAINS_KEY, JSON_APPEND, JSON_REMOVE, JSON_REPLACE, JSON_SET)

71180/17975709 JSON_CONTAINS_KEY founds not existent key in the array element

71267/18020724 buffer underrun in JParser::parse_pair

71264/18022761 No uninstall information in the README

71261/18023002
References in README missing

Article 0

$
0
0

semi-confused about semi-relational

I’m reading a 2012 google paper about Spanner.  I saw Spanner described somewhere as “semi-relational” and wanted to read more.  The paper I’m reading is “Spanner: Google’s Globally-Distributed Database” .

Early on, in page 4, is this paragraph:

“Spanner’s data model is not purely relational, in that  rows must have names. More precisely, every table is required to have an ordered set of one or more primary-key columns. This requirement is where Spanner still looks like a key-value store: the primary keys form the name for a row, and each table defines a mapping from the primary-key columns to the non-primary-key columns.  A row has existence only if some value (even if it is NULL) is defined for the row’s keys. Imposing this structure is useful because it lets applications control data locality through their choices of keys.”

This made no sense to me.  It’s not purely relational because every table needs a primary key?  In relational theory, every relation is required to have at least one candidate key.  Is this confusion between “logical” relational theory and current implementations that allow duplicate rows in tables?  Maybe because it’s an *ordered* set, is that the point?

To me, the not-relational part to me sounds like the fact that primary keys can include NULLs.

Or are they really referring to the fact that data are grouped somewhat hierarchically?  (As explained later in the paper.)  That would make more sense to me.

Anyway, those first three sentences confuse me.  But I’m new to a lot of this.  I’m just a simple caveman.  Your modern ways confuse me.  I’m not arguing that Spanner is purely relational, just saying that I don’t get those first three sentences.  Maybe someone can explain them to me.


Viewing all 352 articles
Browse latest View live


Latest Images

<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>