您当前的位置： 首页 >  sql

实践练习四（必选）：迁移 MySQL 数据到 OceanBase 集群

发布时间：2022-01-11 19:35:56 ，浏览量：0

  • OceanBase Docker安装体验：https://www.xmmup.com/oceanbase-dockeranzhuangtiyan.html
  • 手动部署 OceanBase 单副本集群：https://www.xmmup.com/shoudongbushu-oceanbase-danfubenjiqun.html
  • 手动部署 OceanBase 三副本集群（在同一个节点）：https://www.xmmup.com/shoudongbushu-oceanbase-sanfubenjiqunzaitongyigejiedian.html
  • 使用OBD部署一个OceanBase 三副本集群并使用Promethues监控OB（在不同节点）：https://www.xmmup.com/shiyongobdbushuyigeoceanbase-sanfubenjiqunbingshiyongpromethuesjiankongobzaibutongjiedian.html
说明

相关连接：

https://open.oceanbase.com/blog/10900159

https://open.oceanbase.com/answer/detail?id=20400030

练习目的

本次练习目的掌握从 MySQL 向 OceanBase 迁移数据的基本方法：mysqldump、datax 、canal 等。

练习条件
  • 有 服务器，内存资源至少 12G*1台 ，部署有 OceanBase 集群。
  • 有 mysql 数据库，有少量业务表和数据（数据量不要求很多）。没有业务表和数据的就推荐使用 sysbench、 tpcc 或者 tpch 的业务数据。
练习内容

请记录并分享下列内容：

  • （必选）使用 mysqldump 将 mysql的表结构和数据同步到 OceanBase 的MySQL 租户中。
  • （必选）使用 datax 配置至少一个表的 MySQL 到 OceanBase 的 MySQL 租户的离线同步。
  • （可选）使用 datax 配置至少一个表的 OceanBase 到 CSV 以及 CSV 到 OceanBase 的离线同步。
  • （可选）使用 canal 配置 MySQL 的增量同步到 OceanBase 的 MySQL 租户。
使用 mysqldump 将 mysql的表结构和数据同步到 OceanBase 的MySQL 租户中

源库：MySQL 5.7.36，IP地址：172.17.0.3，数据库：sbtest，10张表，每张表10万行数据

目标库：OB 3.1.1，IP地址：172.17.0.2，数据库：sbtest，租户：obmysql

使用sysbench给mysql环境生成数据 
[root@docker35 ~]# mysql -uroot -plhr -h172.17.0.3 mysql: [Warning] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 2 Server version: 5.7.36-log MySQL Community Server (GPL) Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> mysql> create database sbtest; Query OK, 1 row affected (0.06 sec) Bye [root@docker35 ~]# sysbench /usr/share/sysbench/oltp_common.lua --time=300 --mysql-host=172.17.0.3 --mysql-port=3306 --mysql-user=root --mysql-password=lhr --mysql-db=sbtest --table-size=100000 --tables=10 --threads=16 --events=999999999   prepare sysbench 1.0.20 (using bundled LuaJIT 2.1.0-beta2) Initializing worker threads... Creating table 'sbtest8'... Creating table 'sbtest1'... Creating table 'sbtest7'... Creating table 'sbtest6'... Creating table 'sbtest5'... Creating table 'sbtest3'... Creating table 'sbtest9'... Creating table 'sbtest4'... Creating table 'sbtest10'... Creating table 'sbtest2'... Inserting 100000 records into 'sbtest7' Inserting 100000 records into 'sbtest2' Inserting 100000 records into 'sbtest3' Inserting 100000 records into 'sbtest9' Inserting 100000 records into 'sbtest5' Inserting 100000 records into 'sbtest6' Inserting 100000 records into 'sbtest10' Inserting 100000 records into 'sbtest1' Inserting 100000 records into 'sbtest4' Inserting 100000 records into 'sbtest8' Creating a secondary index on 'sbtest2'... Creating a secondary index on 'sbtest7'... Creating a secondary index on 'sbtest3'... Creating a secondary index on 'sbtest9'... Creating a secondary index on 'sbtest10'... Creating a secondary index on 'sbtest5'... Creating a secondary index on 'sbtest6'... Creating a secondary index on 'sbtest8'... Creating a secondary index on 'sbtest1'... Creating a secondary index on 'sbtest4'... [root@docker35 ~]# mysql -uroot -plhr -h172.17.0.3 mysql: [Warning] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 36 Server version: 5.7.36-log MySQL Community Server (GPL) Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> use sbtest; Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A Database changed
mysql> show tables; +------------------+ | Tables_in_sbtest | +------------------+ | sbtest1 | | sbtest10 | | sbtest2 | | sbtest3 | | sbtest4 | | sbtest5 | | sbtest6 | | sbtest7 | | sbtest8 | | sbtest9 | +------------------+ 10 rows in set (0.00 sec) mysql> select count(*) from sbtest1; +----------+ | count(*) | +----------+ | 100000 | +----------+ 1 row in set (0.04 sec) mysql> SELECT TABLE_NAME,DATA_LENGTH,INDEX_LENGTH,(DATA_LENGTH+INDEX_LENGTH) as length,TABLE_ROWS,concat(round((DATA_LENGTH+INDEX_LENGTH)/1024/1024,2), 'MB') as total_size -> FROM information_schema.TABLES -> WHERE TABLE_SCHEMA='sbtest' -> order by length desc -> ; +------------+-------------+--------------+----------+------------+------------+ | TABLE_NAME | DATA_LENGTH | INDEX_LENGTH | length | TABLE_ROWS | total_size | +------------+-------------+--------------+----------+------------+------------+ | sbtest6 | 22593536 | 1589248 | 24182784 | 98712 | 23.06MB | | sbtest5 | 22593536 | 1589248 | 24182784 | 98712 | 23.06MB | | sbtest4 | 22593536 | 1589248 | 24182784 | 98712 | 23.06MB | | sbtest3 | 22593536 | 1589248 | 24182784 | 98712 | 23.06MB | | sbtest2 | 22593536 | 1589248 | 24182784 | 98712 | 23.06MB | | sbtest10 | 22593536 | 1589248 | 24182784 | 98712 | 23.06MB | | sbtest1 | 22593536 | 1589248 | 24182784 | 98712 | 23.06MB | | sbtest9 | 22593536 | 1589248 | 24182784 | 98712 | 23.06MB | | sbtest8 | 22593536 | 1589248 | 24182784 | 98712 | 23.06MB | | sbtest7 | 22593536 | 1589248 | 24182784 | 98712 | 23.06MB | +------------+-------------+--------------+----------+------------+------------+ 10 rows in set (0.01 sec)
使用mysqldump导出 
[root@docker35 soft]# mysqldump -uroot -plhr -h172.17.0.3 -P3306 --single-transaction --hex-blob --routines --events --triggers --master-data=2 --set-gtid-purged=OFF --databases sbtest --default-character-set=utf8  --max_allowed_packet=512M > sbtest.sql mysqldump: [Warning] Using a password on the command line interface can be insecure. [root@docker35 soft]# ll -h total 315M -rw-r--r-- 1 root root 7.9M Nov  5 09:10 obproxy-3.2.0-1.el7.x86_64.rpm -rw-r--r-- 1 root root  47M Dec 31 17:57 oceanbase-ce-3.1.2-10000392021123010.el7.x86_64.rpm -rw-r--r-- 1 root root 156K Dec 31 19:40 oceanbase-ce-libs-3.1.2-10000392021123010.el7.x86_64.rpm -rw-r--r-- 1 root root  72M Dec 18 06:25 prometheus-2.32.1.linux-amd64.tar.gz -rw-r--r-- 1 root root 189M Jan 10 15:38 sbtest.sql
使用source命令导入数据

OB的环境参考：https://www.xmmup.com/oceanbase-dockeranzhuangtiyan.html 

[admin@0209306f10bf tmp]$  obclient -h 127.1 -uroot@obmysql#obce-single -P2883 -p -c -A test Enter password: 
Welcome to the OceanBase. Commands end with ; or \g. Your MySQL connection id is 10 Server version: 5.6.25 OceanBase 3.1.1 (r4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e) (Built Oct 21 2021 10:52:05) Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MySQL [test]> show databases; +--------------------+ | Database | +--------------------+ | oceanbase | | information_schema | | mysql | | test | | lhrdb | +--------------------+ 5 rows in set (0.005 sec) MySQL [test]> source /tmp/sbtest.sql Query OK, 0 rows affected (0.001 sec) ERROR 1064 (42000) at line 2 in file: '/tmp/sbtest.sql': You have an error in your SQL syntax; check the manual that corresponds to your OceanBase version for the right syntax to use near '--' at line 1 Query OK, 0 rows affected (0.001 sec) Query OK, 5258 rows affected (0.478 sec) Records: 5258 Duplicates: 0 Warnings: 0 ..... Query OK, 0 rows affected (0.001 sec) Query OK, 5258 rows affected (0.456 sec) Records: 5258 Duplicates: 0 Warnings: 0 Query OK, 5258 rows affected (0.478 sec) Records: 5258 Duplicates: 0 Warnings: 0 MySQL [sbtest]> MySQL [sbtest]> MySQL [sbtest]> show databases; +--------------------+ | Database | +--------------------+ | oceanbase | | information_schema | | mysql | | test | | lhrdb | | sbtest | +--------------------+ 6 rows in set (0.006 sec) MySQL [sbtest]> use sbtest; Database changed
MySQL [sbtest]> show tables; +------------------+ | Tables_in_sbtest | +------------------+ | sbtest1 | | sbtest10 | | sbtest2 | | sbtest3 | | sbtest4 | | sbtest5 | | sbtest6 | | sbtest7 | | sbtest8 | | sbtest9 | +------------------+ 10 rows in set (0.004 sec) MySQL [sbtest]> select count(1) from sbtest1; +----------+ | count(1) | +----------+ | 100000 | +----------+ 1 row in set (0.375 sec)
使用datax (离线)从 MySQL 同步表数据到 OceanBase

使用 datax 配置至少一个表的 MySQL 到 OceanBase 的 MySQL 租户的离线同步。

部署DATAX软件

有关dataX更多内容请参考：https://www.xmmup.com/alikaiyuanetlgongjuzhidataxhedatax-webjieshao.html 

-- 大小：820MB wget http://datax-opensource.oss-cn-hangzhou.aliyuncs.com/datax.tar.gz tar -xf datax.tar.gz -C /usr/local/ cd /usr/local/datax # 删除datax中的隐藏文件 find /usr/local/datax/plugin -name ".*" | xargs rm -f


yum install -y python2 java -- 生成模板文件 python2 /usr/local/datax/bin/datax.py -r mysqlreader -w oceanbasev10writer > /usr/local/datax/job/my2ob.json # 根据实际环境修改 vi /usr/local/datax/job/my2ob.json
修改同步作业文件 
cat > /usr/local/datax/job/my2ob.json <<"EOF" { "job": { "setting": { "speed": { "channel": 2, }, "errorLimit": { "record": 10 }
        }, "content": [ { "reader": { "name": "mysqlreader", "parameter": { "username": "root", "password": "lhr", "splitPk": "id", "column": ["*"], "connection": [ { "jdbcUrl": [ "jdbc:mysql://172.17.0.3:3306/sbtest" ], "table": [ "sbtest1" ] } ] }
                }, "writer": { "name": "oceanbasev10writer", "parameter": { "writeMode": "insert", "username": "root", "password": "lhr", "writerThreadCount": 5, "column": [ "*" ], "connection": [ { "jdbcUrl": "||_dsc_ob10_dsc_||obce-single:obmysql||_dsc_ob10_dsc_||jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8", "table": [ "sbtest1" ] } ] }
                }
            } ] }
}
EOF
启动同步作业 
python2 /usr/local/datax/bin/datax.py /usr/local/datax/job/my2ob.json

日志： 

[admin@0209306f10bf ~]$ python2 /usr/local/datax/bin/datax.py /usr/local/datax/job/my2ob.json

DataX (DATAX-OPENSOURCE-3.0), From Alibaba ! Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved. 2022-01-10 09:04:59.884 [main] INFO  VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl 2022-01-10 09:04:59.898 [main] INFO Engine - the machine info => osInfo: Red Hat, Inc. 1.8 25.312-b07
        jvmInfo:        Linux amd64 3.10.0-1127.10.1.el7.x86_64
        cpu num: 16 totalPhysicalMemory: -0.00G
        freePhysicalMemory: -0.00G
        maxFileDescriptorCount: -1 currentOpenFileDescriptorCount: -1 GC Names [PS MarkSweep, PS Scavenge] MEMORY_NAME | allocation_size | init_size                      
        PS Eden Space | 256.00MB | 256.00MB                       
        Code Cache | 240.00MB | 2.44MB                         
        Compressed Class Space | 1,024.00MB | 0.00MB                         
        PS Survivor Space | 42.50MB | 42.50MB                        
        PS Old Gen | 683.00MB | 683.00MB                       
        Metaspace | -0.00MB | 0.00MB 2022-01-10 09:04:59.935 [main] INFO Engine - { "content":[ { "reader":{ "name":"mysqlreader", "parameter":{ "column":[ "*" ], "connection":[ { "jdbcUrl":[ "jdbc:mysql://172.17.0.3:3306/sbtest" ], "table":[ "sbtest1" ] } ], "password":"***", "splitPk":"id", "username":"root" }
                        }, "writer":{ "name":"oceanbasev10writer", "parameter":{ "column":[ "*" ], "connection":[ { "jdbcUrl":"||_dsc_ob10_dsc_||obce-single:obmysql||_dsc_ob10_dsc_||jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8", "table":[ "sbtest1" ] } ], "password":"***", "username":"root", "writeMode":"insert", "writerThreadCount":5 }
                        }
                } ], "setting":{ "errorLimit":{ "record":10 }, "speed":{ "channel":2 }
        }
} 2022-01-10 09:04:59.963 [main] WARN Engine - prioriy set to 0, because NumberFormatException, the value is: null 2022-01-10 09:04:59.969 [main] INFO  PerfTrace - PerfTrace traceId=job_-1, isEnable=false, priority=0 2022-01-10 09:04:59.994 [main] INFO  JobContainer - DataX jobContainer starts job. 2022-01-10 09:04:59.999 [main] INFO  JobContainer - Set jobId = 0 2022-01-10 09:05:00.632 [job-0] INFO  OriginalConfPretreatmentUtil - Available jdbcUrl:jdbc:mysql://172.17.0.3:3306/sbtest?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true. 2022-01-10 09:05:00.634 [job-0] WARN  OriginalConfPretreatmentUtil - 您的配置文件中的列配置存在一定的风险. 因为您未配置读取数据库表的列，当您的表字段个数、类型有变动时，可能影响任务正确性甚至会运行出错。请检查您的配置并作出修改. 2022-01-10 09:05:00.659 [job-0] INFO  DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:00.686 [job-0] INFO  DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8 2022-01-10 09:05:01.073 [job-0] INFO  DbUtils - value for query [SHOW VARIABLES LIKE 'ob_compatibility_mode'] is [MYSQL] 2022-01-10 09:05:01.086 [job-0] INFO  DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:01.117 [job-0] INFO  DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2022-01-10 09:05:01.136 [job-0] INFO  OriginalConfPretreatmentUtil - table:[sbtest1] all columns:[ id,k,c,pad ]. 2022-01-10 09:05:01.164 [job-0] WARN  OriginalConfPretreatmentUtil - 您的配置文件中的列配置信息存在风险. 因为您配置的写入数据库表的列为*，当您的表字段个数、类型有变动时，可能影响任务正确性甚至会运行出错。请检查您的配置并作出修改. 2022-01-10 09:05:01.167 [job-0] INFO  OriginalConfPretreatmentUtil - Write data [ insert INTO %s (id,k,c,pad) VALUES(?,?,?,?) ], which jdbcUrl like:[||_dsc_ob10_dsc_||obce-single:obmysql||_dsc_ob10_dsc_||jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true] 2022-01-10 09:05:01.168 [job-0] INFO  JobContainer - jobContainer starts to do prepare ... 2022-01-10 09:05:01.169 [job-0] INFO  JobContainer - DataX Reader.Job [mysqlreader] do prepare work . 2022-01-10 09:05:01.169 [job-0] INFO  JobContainer - DataX Writer.Job [oceanbasev10writer] do prepare work . 2022-01-10 09:05:01.171 [job-0] INFO  DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:01.171 [job-0] INFO  DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2022-01-10 09:05:01.191 [job-0] INFO  DbUtils - value for query [show variables like 'version'] is [3.1.1] 2022-01-10 09:05:01.206 [job-0] INFO  JobContainer - jobContainer starts to do split ... 2022-01-10 09:05:01.206 [job-0] INFO  JobContainer - Job set Channel-Number to 2 channels. 2022-01-10 09:05:01.226 [job-0] INFO  SingleTableSplitUtil - split pk [sql=SELECT MIN(id),MAX(id) FROM sbtest1] is running... 2022-01-10 09:05:01.256 [job-0] INFO  SingleTableSplitUtil - After split(), allQuerySql=[ select * from sbtest1 where (1 <= id AND id < 10001) select * from sbtest1 where (10001 <= id AND id < 20001) select * from sbtest1 where (20001 <= id AND id < 30001) select * from sbtest1 where (30001 <= id AND id < 40001) select * from sbtest1 where (40001 <= id AND id < 50001) select * from sbtest1 where (50001 <= id AND id < 60001) select * from sbtest1 where (60001 <= id AND id < 70001) select * from sbtest1 where (70001 <= id AND id < 80001) select * from sbtest1 where (80001 <= id AND id < 90001) select * from sbtest1 where (90001 <= id AND id <= 100000) select * from sbtest1 where id IS NULL ]. 2022-01-10 09:05:01.256 [job-0] INFO  JobContainer - DataX Reader.Job [mysqlreader] splits to [11] tasks. 2022-01-10 09:05:01.260 [job-0] INFO  JobContainer - DataX Writer.Job [oceanbasev10writer] splits to [11] tasks. 2022-01-10 09:05:01.300 [job-0] INFO  JobContainer - jobContainer starts to do schedule ... 2022-01-10 09:05:01.315 [job-0] INFO  JobContainer - Scheduler starts [1] taskGroups. 2022-01-10 09:05:01.320 [job-0] INFO  JobContainer - Running by standalone Mode. 2022-01-10 09:05:01.337 [taskGroup-0] INFO  TaskGroupContainer - taskGroupId=[0] start [2] channels for [11] tasks. 2022-01-10 09:05:01.358 [taskGroup-0] INFO  Channel - Channel set byte_speed_limit to -1, No bps activated. 2022-01-10 09:05:01.358 [taskGroup-0] INFO  Channel - Channel set record_speed_limit to -1, No tps activated. 2022-01-10 09:05:01.373 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started 2022-01-10 09:05:01.378 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[1] attemptCount[1] is started 2022-01-10 09:05:01.379 [0-0-0-writer] INFO  OceanBaseV10Writer$Task - tableNumber:1,writerTask Class:com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask 2022-01-10 09:05:01.379 [0-0-1-writer] INFO  OceanBaseV10Writer$Task - tableNumber:1,writerTask Class:com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask 2022-01-10 09:05:01.380 [0-0-0-reader] INFO  CommonRdbmsReader$Task - Begin to read record by Sql: [select * from sbtest1 where (1 <= id AND id < 10001) ] jdbcUrl:[jdbc:mysql://172.17.0.3:3306/sbtest?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2022-01-10 09:05:01.380 [0-0-1-reader] INFO  CommonRdbmsReader$Task - Begin to read record by Sql: [select * from sbtest1 where (10001 <= id AND id < 20001) ] jdbcUrl:[jdbc:mysql://172.17.0.3:3306/sbtest?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2022-01-10 09:05:01.382 [0-0-1-writer] INFO  ConcurrentTableWriterTask - configure url is unavailable, use obclient for connections. 2022-01-10 09:05:01.382 [0-0-0-writer] INFO  ConcurrentTableWriterTask - configure url is unavailable, use obclient for connections. 2022-01-10 09:05:01.403 [0-0-1-writer] INFO  ConcurrentTableWriterTask - Disable partition calculation feature. 2022-01-10 09:05:01.431 [0-0-0-writer] INFO  ConcurrentTableWriterTask - Disable partition calculation feature. 2022-01-10 09:05:01.449 [0-0-1-writer] INFO  CommonRdbmsWriter$Task - write mode: update 2022-01-10 09:05:01.455 [0-0-1-writer] INFO  CommonRdbmsWriter$Task - Skip columns: ID, 2022-01-10 09:05:01.456 [0-0-1-writer] INFO  ConcurrentTableWriterTask - writeRecordSql :INSERT INTO sbtest1 (id,k,c,pad) VALUES(?,?,?,?) ON DUPLICATE KEY UPDATE k=VALUES(k),c=VALUES(c),pad=VALUES(pad) 2022-01-10 09:05:01.464 [0-0-0-writer] INFO  CommonRdbmsWriter$Task - write mode: update 2022-01-10 09:05:01.464 [0-0-1-writer] INFO  DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:01.464 [0-0-1-writer] INFO  DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2022-01-10 09:05:01.469 [0-0-0-writer] INFO  CommonRdbmsWriter$Task - Skip columns: ID, 2022-01-10 09:05:01.470 [0-0-0-writer] INFO  ConcurrentTableWriterTask - writeRecordSql :INSERT INTO sbtest1 (id,k,c,pad) VALUES(?,?,?,?) ON DUPLICATE KEY UPDATE k=VALUES(k),c=VALUES(c),pad=VALUES(pad) 2022-01-10 09:05:01.482 [0-0-0-writer] INFO  DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:01.482 [0-0-1-writer] ERROR ConcurrentTableWriterTask - partCalculator is null 2022-01-10 09:05:01.483 [0-0-0-writer] INFO  DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2022-01-10 09:05:01.483 [0-0-1-writer] INFO  ConcurrentTableWriterTask - start 1 insert task. 2022-01-10 09:05:01.500 [0-0-0-writer] ERROR ConcurrentTableWriterTask - partCalculator is null 2022-01-10 09:05:01.502 [0-0-0-writer] INFO  ConcurrentTableWriterTask - start 1 insert task. 2022-01-10 09:05:01.512 [0-0-1-writer] INFO  ConcurrentTableWriterTask - start 2 insert task. 2022-01-10 09:05:01.527 [0-0-0-writer] INFO  ConcurrentTableWriterTask - start 2 insert task. 2022-01-10 09:05:01.543 [0-0-1-writer] INFO  ConcurrentTableWriterTask - start 3 insert task. 2022-01-10 09:05:01.562 [0-0-0-writer] INFO  ConcurrentTableWriterTask - start 3 insert task. 2022-01-10 09:05:01.575 [0-0-1-writer] INFO  ConcurrentTableWriterTask - start 4 insert task. 2022-01-10 09:05:01.589 [0-0-0-writer] INFO  ConcurrentTableWriterTask - start 4 insert task. 2022-01-10 09:05:01.615 [0-0-1-writer] INFO  ConcurrentTableWriterTask - start 5 insert task. 2022-01-10 09:05:01.627 [0-0-0-writer] INFO  ConcurrentTableWriterTask - start 5 insert task. 2022-01-10 09:05:01.659 [0-0-1-writer] INFO  DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:01.659 [0-0-1-writer] INFO  DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2022-01-10 09:05:01.673 [0-0-0-writer] INFO  DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:01.673 [0-0-0-writer] INFO  DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2022-01-10 09:05:01.677 [0-0-1-writer] INFO  ColumnMetaCache - fetch columnMeta of table sbtest1 success 2022-01-10 09:05:01.773 [0-0-1-writer] INFO  CommonRdbmsWriter$Task - isMemstoreFull=false 2022-01-10 09:05:01.813 [0-0-0-writer] INFO  CommonRdbmsWriter$Task - isMemstoreFull=false 2022-01-10 09:05:02.043 [0-0-1-reader] INFO  CommonRdbmsReader$Task - Finished read record by Sql: [select * from sbtest1 where (10001 <= id AND id < 20001) ] jdbcUrl:[jdbc:mysql://172.17.0.3:3306/sbtest?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2022-01-10 09:05:02.131 [0-0-0-reader] INFO  CommonRdbmsReader$Task - Finished read record by Sql: [select * from sbtest1 where (1 <= id AND id < 10001) ] jdbcUrl:[jdbc:mysql://172.17.0.3:3306/sbtest?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2022-01-10 09:05:02.186 [0-0-0-writer] INFO  ConcurrentTableWriterTask - ConcurrentTableWriter has put all task in queue, queueSize = 0, total = 5, finished = 0 2022-01-10 09:05:02.260 [0-0-1-writer] INFO  ConcurrentTableWriterTask - ConcurrentTableWriter has put all task in queue, queueSize = 0, total = 5, finished = 0 2022-01-10 09:05:02.781 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[0] is successed, used[1409]ms 2022-01-10 09:05:02.782 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[1] is successed, used[1405]ms 2022-01-10 09:05:02.784 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[2] attemptCount[1] is started 2022-01-10 09:05:02.784 [0-0-2-writer] INFO  OceanBaseV10Writer$Task - tableNumber:1,writerTask Class:com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask 2022-01-10 09:05:02.785 [0-0-2-reader] INFO  CommonRdbmsReader$Task - Begin to read record by Sql: [select * from sbtest1 where (20001 <= id AND id < 30001) ] jdbcUrl:[jdbc:mysql://172.17.0.3:3306/sbtest?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2022-01-10 09:05:02.785 [0-0-2-writer] INFO  ConcurrentTableWriterTask - configure url is unavailable, use obclient for connections. 2022-01-10 09:05:02.786 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[3] attemptCount[1] is started 2022-01-10 09:05:02.812 [0-0-3-writer] INFO  OceanBaseV10Writer$Task - tableNumber:1,writerTask Class:com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask 2022-01-10 09:05:02.812 [0-0-3-reader] INFO  CommonRdbmsReader$Task - Begin to read record by Sql: [select * from sbtest1 where (30001 <= id AND id < 40001) ] jdbcUrl:[jdbc:mysql://172.17.0.3:3306/sbtest?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2022-01-10 09:05:02.813 [0-0-3-writer] INFO  ConcurrentTableWriterTask - configure url is unavailable, use obclient for connections. 2022-01-10 09:05:02.820 [0-0-2-writer] INFO  ConcurrentTableWriterTask - Disable partition calculation feature. 2022-01-10 09:05:02.837 [0-0-3-writer] INFO  ConcurrentTableWriterTask - Disable partition calculation feature. 2022-01-10 09:05:02.845 [0-0-2-writer] INFO  CommonRdbmsWriter$Task - write mode: update 2022-01-10 09:05:02.849 [0-0-2-writer] INFO  CommonRdbmsWriter$Task - Skip columns: ID, 2022-01-10 09:05:02.850 [0-0-2-writer] INFO  ConcurrentTableWriterTask - writeRecordSql :INSERT INTO sbtest1 (id,k,c,pad) VALUES(?,?,?,?) ON DUPLICATE KEY UPDATE k=VALUES(k),c=VALUES(c),pad=VALUES(pad) 2022-01-10 09:05:02.854 [0-0-3-writer] INFO  CommonRdbmsWriter$Task - write mode: update 2022-01-10 09:05:02.854 [0-0-2-writer] INFO  DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:02.854 [0-0-2-writer] INFO  DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2022-01-10 09:05:02.857 [0-0-3-writer] INFO  CommonRdbmsWriter$Task - Skip columns: ID, 2022-01-10 09:05:02.858 [0-0-3-writer] INFO  ConcurrentTableWriterTask - writeRecordSql :INSERT INTO sbtest1 (id,k,c,pad) VALUES(?,?,?,?) ON DUPLICATE KEY UPDATE k=VALUES(k),c=VALUES(c),pad=VALUES(pad) 2022-01-10 09:05:02.863 [0-0-3-writer] INFO  DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:02.863 [0-0-2-writer] ERROR ConcurrentTableWriterTask - partCalculator is null 2022-01-10 09:05:02.863 [0-0-3-writer] INFO  DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2022-01-10 09:05:02.863 [0-0-2-writer] INFO  ConcurrentTableWriterTask - start 1 insert task. 2022-01-10 09:05:02.874 [0-0-3-writer] ERROR ConcurrentTableWriterTask - partCalculator is null 2022-01-10 09:05:02.874 [0-0-3-writer] INFO  ConcurrentTableWriterTask - start 1 insert task. 2022-01-10 09:05:02.884 [0-0-2-writer] INFO  ConcurrentTableWriterTask - start 2 insert task. 2022-01-10 09:05:02.893 [0-0-3-writer] INFO  ConcurrentTableWriterTask - start 2 insert task. 2022-01-10 09:05:02.917 [0-0-2-writer] INFO  ConcurrentTableWriterTask - start 3 insert task. 2022-01-10 09:05:02.925 [0-0-3-writer] INFO  ConcurrentTableWriterTask - start 3 insert task. 2022-01-10 09:05:02.933 [0-0-2-writer] INFO  ConcurrentTableWriterTask - start 4 insert task. 2022-01-10 09:05:02.941 [0-0-3-writer] INFO  ConcurrentTableWriterTask - start 4 insert task. 2022-01-10 09:05:02.949 [0-0-2-writer] INFO  ConcurrentTableWriterTask - start 5 insert task. 2022-01-10 09:05:02.957 [0-0-3-writer] INFO  ConcurrentTableWriterTask - start 5 insert task. 2022-01-10 09:05:02.972 [0-0-2-writer] INFO  DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:02.973 [0-0-2-writer] INFO  DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2022-01-10 09:05:02.980 [0-0-3-writer] INFO  DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:02.980 [0-0-3-writer] INFO  DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2022-01-10 09:05:03.000 [0-0-2-writer] INFO  CommonRdbmsWriter$Task - isMemstoreFull=false 2022-01-10 09:05:03.011 [0-0-3-writer] INFO  CommonRdbmsWriter$Task - isMemstoreFull=false 2022-01-10 09:05:03.091 [0-0-2-reader] INFO  CommonRdbmsReader$Task - Finished read record by Sql: [select * from sbtest1 where (20001 <= id AND id < 30001) ] jdbcUrl:[jdbc:mysql://172.17.0.3:3306/sbtest?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2022-01-10 09:05:03.099 [0-0-3-reader] INFO  CommonRdbmsReader$Task - Finished read record by Sql: [select * from sbtest1 where (30001 <= id AND id < 40001) ] jdbcUrl:[jdbc:mysql://172.17.0.3:3306/sbtest?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2022-01-10 09:05:03.117 [0-0-2-writer] INFO  ConcurrentTableWriterTask - ConcurrentTableWriter has put all task in queue, queueSize = 0, total = 5, finished = 0 2022-01-10 09:05:03.126 [0-0-3-writer] INFO  ConcurrentTableWriterTask - ConcurrentTableWriter has put all task in queue, queueSize = 0, total = 5, finished = 0 2022-01-10 09:05:03.589 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[2] is successed, used[805]ms 2022-01-10 09:05:03.592 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[4] attemptCount[1] is started 2022-01-10 09:05:03.592 [0-0-4-writer] INFO  OceanBaseV10Writer$Task - tableNumber:1,writerTask Class:com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask 2022-01-10 09:05:03.593 [0-0-4-reader] INFO  CommonRdbmsReader$Task - Begin to read record by Sql: [select * from sbtest1 where (40001 <= id AND id < 50001) ] jdbcUrl:[jdbc:mysql://172.17.0.3:3306/sbtest?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2022-01-10 09:05:03.627 [0-0-4-writer] INFO  ConcurrentTableWriterTask - configure url is unavailable, use obclient for connections. 2022-01-10 09:05:03.639 [0-0-4-writer] INFO  ConcurrentTableWriterTask - Disable partition calculation feature. 2022-01-10 09:05:03.647 [0-0-4-writer] INFO  CommonRdbmsWriter$Task - write mode: update 2022-01-10 09:05:03.651 [0-0-4-writer] INFO  CommonRdbmsWriter$Task - Skip columns: ID, 2022-01-10 09:05:03.652 [0-0-4-writer] INFO  ConcurrentTableWriterTask - writeRecordSql :INSERT INTO sbtest1 (id,k,c,pad) VALUES(?,?,?,?) ON DUPLICATE KEY UPDATE k=VALUES(k),c=VALUES(c),pad=VALUES(pad) 2022-01-10 09:05:03.653 [0-0-4-writer] INFO  DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:03.653 [0-0-4-writer] INFO  DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2022-01-10 09:05:03.661 [0-0-4-writer] ERROR ConcurrentTableWriterTask - partCalculator is null 2022-01-10 09:05:03.661 [0-0-4-writer] INFO  ConcurrentTableWriterTask - start 1 insert task. 2022-01-10 09:05:03.669 [0-0-4-writer] INFO  ConcurrentTableWriterTask - start 2 insert task. 2022-01-10 09:05:03.690 [0-0-4-writer] INFO  ConcurrentTableWriterTask - start 3 insert task. 2022-01-10 09:05:03.699 [0-0-4-writer] INFO  ConcurrentTableWriterTask - start 4 insert task. 2022-01-10 09:05:03.706 [0-0-4-writer] INFO  ConcurrentTableWriterTask - start 5 insert task. 2022-01-10 09:05:03.725 [0-0-4-writer] INFO  DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:03.726 [0-0-4-writer] INFO  DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2022-01-10 09:05:03.726 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[3] is successed, used[940]ms 2022-01-10 09:05:03.729 [0-0-5-writer] INFO  OceanBaseV10Writer$Task - tableNumber:1,writerTask Class:com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask 2022-01-10 09:05:03.729 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[5] attemptCount[1] is started 2022-01-10 09:05:03.729 [0-0-5-reader] INFO  CommonRdbmsReader$Task - Begin to read record by Sql: [select * from sbtest1 where (50001 <= id AND id < 60001) ] jdbcUrl:[jdbc:mysql://172.17.0.3:3306/sbtest?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2022-01-10 09:05:03.730 [0-0-5-writer] INFO  ConcurrentTableWriterTask - configure url is unavailable, use obclient for connections. 2022-01-10 09:05:03.753 [0-0-5-writer] INFO  ConcurrentTableWriterTask - Disable partition calculation feature. 2022-01-10 09:05:03.764 [0-0-4-writer] INFO  CommonRdbmsWriter$Task - isMemstoreFull=false 2022-01-10 09:05:03.768 [0-0-5-writer] INFO  CommonRdbmsWriter$Task - write mode: update 2022-01-10 09:05:03.772 [0-0-5-writer] INFO  CommonRdbmsWriter$Task - Skip columns: ID, 2022-01-10 09:05:03.772 [0-0-5-writer] INFO  ConcurrentTableWriterTask - writeRecordSql :INSERT INTO sbtest1 (id,k,c,pad) VALUES(?,?,?,?) ON DUPLICATE KEY UPDATE k=VALUES(k),c=VALUES(c),pad=VALUES(pad) 2022-01-10 09:05:03.773 [0-0-5-writer] INFO  DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:03.774 [0-0-5-writer] INFO  DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2022-01-10 09:05:03.790 [0-0-5-writer] ERROR ConcurrentTableWriterTask - partCalculator is null 2022-01-10 09:05:03.790 [0-0-5-writer] INFO  ConcurrentTableWriterTask - start 1 insert task. 2022-01-10 09:05:03.810 [0-0-5-writer] INFO  ConcurrentTableWriterTask - start 2 insert task. 2022-01-10 09:05:03.839 [0-0-4-reader] INFO  CommonRdbmsReader$Task - Finished read record by Sql: [select * from sbtest1 where (40001 <= id AND id < 50001) ] jdbcUrl:[jdbc:mysql://172.17.0.3:3306/sbtest?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2022-01-10 09:05:03.848 [0-0-5-writer] INFO  ConcurrentTableWriterTask - start 3 insert task. 2022-01-10 09:05:03.857 [0-0-5-writer] INFO  ConcurrentTableWriterTask - start 4 insert task. 2022-01-10 09:05:03.866 [0-0-5-writer] INFO  ConcurrentTableWriterTask - start 5 insert task. 2022-01-10 09:05:03.875 [0-0-5-writer] INFO  DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:03.876 [0-0-5-writer] INFO  DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2022-01-10 09:05:03.906 [0-0-5-writer] INFO  CommonRdbmsWriter$Task - isMemstoreFull=false 2022-01-10 09:05:04.034 [0-0-5-reader] INFO  CommonRdbmsReader$Task - Finished read record by Sql: [select * from sbtest1 where (50001 <= id AND id < 60001) ] jdbcUrl:[jdbc:mysql://172.17.0.3:3306/sbtest?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2022-01-10 09:05:04.051 [0-0-5-writer] INFO  ConcurrentTableWriterTask - ConcurrentTableWriter has put all task in queue, queueSize = 0, total = 5, finished = 0 2022-01-10 09:05:04.061 [0-0-4-writer] INFO  ConcurrentTableWriterTask - ConcurrentTableWriter has put all task in queue, queueSize = 0, total = 5, finished = 0 2022-01-10 09:05:04.531 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[4] is successed, used[939]ms 2022-01-10 09:05:04.531 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[5] is successed, used[803]ms 2022-01-10 09:05:04.533 [0-0-6-writer] INFO  OceanBaseV10Writer$Task - tableNumber:1,writerTask Class:com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask 2022-01-10 09:05:04.533 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[6] attemptCount[1] is started 2022-01-10 09:05:04.534 [0-0-6-reader] INFO  CommonRdbmsReader$Task - Begin to read record by Sql: [select * from sbtest1 where (60001 <= id AND id < 70001) ] jdbcUrl:[jdbc:mysql://172.17.0.3:3306/sbtest?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2022-01-10 09:05:04.534 [0-0-6-writer] INFO  ConcurrentTableWriterTask - configure url is unavailable, use obclient for connections. 2022-01-10 09:05:04.537 [0-0-7-writer] INFO  OceanBaseV10Writer$Task - tableNumber:1,writerTask Class:com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask 2022-01-10 09:05:04.537 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[7] attemptCount[1] is started 2022-01-10 09:05:04.537 [0-0-7-reader] INFO  CommonRdbmsReader$Task - Begin to read record by Sql: [select * from sbtest1 where (70001 <= id AND id < 80001) ] jdbcUrl:[jdbc:mysql://172.17.0.3:3306/sbtest?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2022-01-10 09:05:04.537 [0-0-7-writer] INFO  ConcurrentTableWriterTask - configure url is unavailable, use obclient for connections. 2022-01-10 09:05:04.545 [0-0-6-writer] INFO  ConcurrentTableWriterTask - Disable partition calculation feature. 2022-01-10 09:05:04.554 [0-0-7-writer] INFO  ConcurrentTableWriterTask - Disable partition calculation feature. 2022-01-10 09:05:04.566 [0-0-6-writer] INFO  CommonRdbmsWriter$Task - write mode: update 2022-01-10 09:05:04.571 [0-0-6-writer] INFO  CommonRdbmsWriter$Task - Skip columns: ID, 2022-01-10 09:05:04.572 [0-0-6-writer] INFO  ConcurrentTableWriterTask - writeRecordSql :INSERT INTO sbtest1 (id,k,c,pad) VALUES(?,?,?,?) ON DUPLICATE KEY UPDATE k=VALUES(k),c=VALUES(c),pad=VALUES(pad) 2022-01-10 09:05:04.576 [0-0-7-writer] INFO  CommonRdbmsWriter$Task - write mode: update 2022-01-10 09:05:04.576 [0-0-6-writer] INFO  DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:04.576 [0-0-6-writer] INFO  DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2022-01-10 09:05:04.580 [0-0-7-writer] INFO  CommonRdbmsWriter$Task - Skip columns: ID, 2022-01-10 09:05:04.580 [0-0-7-writer] INFO  ConcurrentTableWriterTask - writeRecordSql :INSERT INTO sbtest1 (id,k,c,pad) VALUES(?,?,?,?) ON DUPLICATE KEY UPDATE k=VALUES(k),c=VALUES(c),pad=VALUES(pad) 2022-01-10 09:05:04.585 [0-0-7-writer] INFO  DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:04.585 [0-0-6-writer] ERROR ConcurrentTableWriterTask - partCalculator is null 2022-01-10 09:05:04.585 [0-0-6-writer] INFO  ConcurrentTableWriterTask - start 1 insert task. 2022-01-10 09:05:04.585 [0-0-7-writer] INFO  DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2022-01-10 09:05:04.595 [0-0-7-writer] ERROR ConcurrentTableWriterTask - partCalculator is null 2022-01-10 09:05:04.595 [0-0-7-writer] INFO  ConcurrentTableWriterTask - start 1 insert task. 2022-01-10 09:05:04.603 [0-0-6-writer] INFO  ConcurrentTableWriterTask - start 2 insert task. 2022-01-10 09:05:04.612 [0-0-7-writer] INFO  ConcurrentTableWriterTask - start 2 insert task. 2022-01-10 09:05:04.620 [0-0-6-writer] INFO  ConcurrentTableWriterTask - start 3 insert task. 2022-01-10 09:05:04.628 [0-0-7-writer] INFO  ConcurrentTableWriterTask - start 3 insert task. 2022-01-10 09:05:04.636 [0-0-6-writer] INFO  ConcurrentTableWriterTask - start 4 insert task. 2022-01-10 09:05:04.643 [0-0-7-writer] INFO  ConcurrentTableWriterTask - start 4 insert task. 2022-01-10 09:05:04.651 [0-0-6-writer] INFO  ConcurrentTableWriterTask - start 5 insert task. 2022-01-10 09:05:04.659 [0-0-7-writer] INFO  ConcurrentTableWriterTask - start 5 insert task. 2022-01-10 09:05:04.673 [0-0-6-writer] INFO  DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:04.673 [0-0-6-writer] INFO  DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2022-01-10 09:05:04.683 [0-0-7-writer] INFO  DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:04.683 [0-0-7-writer] INFO  DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2022-01-10 09:05:04.711 [0-0-6-writer] INFO  CommonRdbmsWriter$Task - isMemstoreFull=false 2022-01-10 09:05:04.719 [0-0-7-writer] INFO  CommonRdbmsWriter$Task - isMemstoreFull=false 2022-01-10 09:05:04.811 [0-0-6-reader] INFO  CommonRdbmsReader$Task - Finished read record by Sql: [select * from sbtest1 where (60001 <= id AND id < 70001) ] jdbcUrl:[jdbc:mysql://172.17.0.3:3306/sbtest?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2022-01-10 09:05:04.822 [0-0-7-reader] INFO  CommonRdbmsReader$Task - Finished read record by Sql: [select * from sbtest1 where (70001 <= id AND id < 80001) ] jdbcUrl:[jdbc:mysql://172.17.0.3:3306/sbtest?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2022-01-10 09:05:04.834 [0-0-6-writer] INFO  ConcurrentTableWriterTask - ConcurrentTableWriter has put all task in queue, queueSize = 0, total = 5, finished = 0 2022-01-10 09:05:04.846 [0-0-7-writer] INFO  ConcurrentTableWriterTask - ConcurrentTableWriter has put all task in queue, queueSize = 0, total = 5, finished = 0 2022-01-10 09:05:05.339 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[7] is successed, used[803]ms 2022-01-10 09:05:05.343 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[8] attemptCount[1] is started 2022-01-10 09:05:05.343 [0-0-8-reader] INFO  CommonRdbmsReader$Task - Begin to read record by Sql: [select * from sbtest1 where (80001 <= id AND id < 90001) ] jdbcUrl:[jdbc:mysql://172.17.0.3:3306/sbtest?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2022-01-10 09:05:05.343 [0-0-8-writer] INFO  OceanBaseV10Writer$Task - tableNumber:1,writerTask Class:com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask 2022-01-10 09:05:05.344 [0-0-8-writer] INFO  ConcurrentTableWriterTask - configure url is unavailable, use obclient for connections. 2022-01-10 09:05:05.352 [0-0-8-writer] INFO  ConcurrentTableWriterTask - Disable partition calculation feature. 2022-01-10 09:05:05.360 [0-0-8-writer] INFO  CommonRdbmsWriter$Task - write mode: update 2022-01-10 09:05:05.364 [0-0-8-writer] INFO  CommonRdbmsWriter$Task - Skip columns: ID, 2022-01-10 09:05:05.365 [0-0-8-writer] INFO  ConcurrentTableWriterTask - writeRecordSql :INSERT INTO sbtest1 (id,k,c,pad) VALUES(?,?,?,?) ON DUPLICATE KEY UPDATE k=VALUES(k),c=VALUES(c),pad=VALUES(pad) 2022-01-10 09:05:05.370 [0-0-8-writer] INFO  DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:05.370 [0-0-8-writer] INFO  DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2022-01-10 09:05:05.381 [0-0-8-writer] ERROR ConcurrentTableWriterTask - partCalculator is null 2022-01-10 09:05:05.381 [0-0-8-writer] INFO  ConcurrentTableWriterTask - start 1 insert task. 2022-01-10 09:05:05.389 [0-0-8-writer] INFO  ConcurrentTableWriterTask - start 2 insert task. 2022-01-10 09:05:05.398 [0-0-8-writer] INFO  ConcurrentTableWriterTask - start 3 insert task. 2022-01-10 09:05:05.408 [0-0-8-writer] INFO  ConcurrentTableWriterTask - start 4 insert task. 2022-01-10 09:05:05.416 [0-0-8-writer] INFO  ConcurrentTableWriterTask - start 5 insert task. 2022-01-10 09:05:05.423 [0-0-8-writer] INFO  DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:05.423 [0-0-8-writer] INFO  DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2022-01-10 09:05:05.444 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[6] is successed, used[910]ms 2022-01-10 09:05:05.446 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[9] attemptCount[1] is started 2022-01-10 09:05:05.446 [0-0-9-writer] INFO  OceanBaseV10Writer$Task - tableNumber:1,writerTask Class:com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask 2022-01-10 09:05:05.446 [0-0-9-reader] INFO  CommonRdbmsReader$Task - Begin to read record by Sql: [select * from sbtest1 where (90001 <= id AND id <= 100000) ] jdbcUrl:[jdbc:mysql://172.17.0.3:3306/sbtest?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2022-01-10 09:05:05.447 [0-0-9-writer] INFO  ConcurrentTableWriterTask - configure url is unavailable, use obclient for connections. 2022-01-10 09:05:05.452 [0-0-8-writer] INFO  CommonRdbmsWriter$Task - isMemstoreFull=false 2022-01-10 09:05:05.456 [0-0-9-writer] INFO  ConcurrentTableWriterTask - Disable partition calculation feature. 2022-01-10 09:05:05.502 [0-0-9-writer] INFO  CommonRdbmsWriter$Task - write mode: update 2022-01-10 09:05:05.506 [0-0-9-writer] INFO  CommonRdbmsWriter$Task - Skip columns: ID, 2022-01-10 09:05:05.506 [0-0-9-writer] INFO  ConcurrentTableWriterTask - writeRecordSql :INSERT INTO sbtest1 (id,k,c,pad) VALUES(?,?,?,?) ON DUPLICATE KEY UPDATE k=VALUES(k),c=VALUES(c),pad=VALUES(pad) 2022-01-10 09:05:05.510 [0-0-9-writer] INFO  DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:05.510 [0-0-9-writer] INFO  DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2022-01-10 09:05:05.524 [0-0-9-writer] ERROR ConcurrentTableWriterTask - partCalculator is null 2022-01-10 09:05:05.524 [0-0-9-writer] INFO  ConcurrentTableWriterTask - start 1 insert task. 2022-01-10 09:05:05.540 [0-0-9-writer] INFO  ConcurrentTableWriterTask - start 2 insert task. 2022-01-10 09:05:05.552 [0-0-8-reader] INFO  CommonRdbmsReader$Task - Finished read record by Sql: [select * from sbtest1 where (80001 <= id AND id < 90001) ] jdbcUrl:[jdbc:mysql://172.17.0.3:3306/sbtest?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2022-01-10 09:05:05.560 [0-0-9-writer] INFO  ConcurrentTableWriterTask - start 3 insert task. 2022-01-10 09:05:05.568 [0-0-8-writer] INFO  ConcurrentTableWriterTask - ConcurrentTableWriter has put all task in queue, queueSize = 0, total = 5, finished = 0 2022-01-10 09:05:05.579 [0-0-9-writer] INFO  ConcurrentTableWriterTask - start 4 insert task. 2022-01-10 09:05:05.601 [0-0-9-writer] INFO  ConcurrentTableWriterTask - start 5 insert task. 2022-01-10 09:05:05.608 [0-0-9-writer] INFO  DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:05.608 [0-0-9-writer] INFO  DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2022-01-10 09:05:05.837 [0-0-9-writer] INFO  CommonRdbmsWriter$Task - isMemstoreFull=false 2022-01-10 09:05:06.462 [0-0-9-reader] INFO  CommonRdbmsReader$Task - Finished read record by Sql: [select * from sbtest1 where (90001 <= id AND id <= 100000) ] jdbcUrl:[jdbc:mysql://172.17.0.3:3306/sbtest?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2022-01-10 09:05:06.483 [0-0-9-writer] INFO  ConcurrentTableWriterTask - ConcurrentTableWriter has put all task in queue, queueSize = 0, total = 5, finished = 0 2022-01-10 09:05:06.827 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[8] is successed, used[1485]ms 2022-01-10 09:05:06.831 [0-0-10-writer] INFO  OceanBaseV10Writer$Task - tableNumber:1,writerTask Class:com.alibaba.datax.plugin.writer.oceanbasev10writer.task.ConcurrentTableWriterTask 2022-01-10 09:05:06.831 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[10] attemptCount[1] is started 2022-01-10 09:05:06.831 [0-0-10-reader] INFO  CommonRdbmsReader$Task - Begin to read record by Sql: [select * from sbtest1 where id IS NULL ] jdbcUrl:[jdbc:mysql://172.17.0.3:3306/sbtest?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2022-01-10 09:05:06.832 [0-0-10-writer] INFO  ConcurrentTableWriterTask - configure url is unavailable, use obclient for connections. 2022-01-10 09:05:06.928 [0-0-10-reader] INFO  CommonRdbmsReader$Task - Finished read record by Sql: [select * from sbtest1 where id IS NULL ] jdbcUrl:[jdbc:mysql://172.17.0.3:3306/sbtest?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]. 2022-01-10 09:05:06.928 [0-0-10-writer] INFO  ConcurrentTableWriterTask - Disable partition calculation feature. 2022-01-10 09:05:07.015 [0-0-10-writer] INFO  CommonRdbmsWriter$Task - write mode: update 2022-01-10 09:05:07.038 [0-0-10-writer] INFO  CommonRdbmsWriter$Task - Skip columns: ID, 2022-01-10 09:05:07.038 [0-0-10-writer] INFO  ConcurrentTableWriterTask - writeRecordSql :INSERT INTO sbtest1 (id,k,c,pad) VALUES(?,?,?,?) ON DUPLICATE KEY UPDATE k=VALUES(k),c=VALUES(c),pad=VALUES(pad) 2022-01-10 09:05:07.039 [0-0-10-writer] INFO  DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:07.040 [0-0-10-writer] INFO  DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2022-01-10 09:05:07.139 [0-0-10-writer] ERROR ConcurrentTableWriterTask - partCalculator is null 2022-01-10 09:05:07.139 [0-0-10-writer] INFO  ConcurrentTableWriterTask - start 1 insert task. 2022-01-10 09:05:07.216 [0-0-10-writer] INFO  ConcurrentTableWriterTask - start 2 insert task. 2022-01-10 09:05:07.247 [0-0-10-writer] INFO  ConcurrentTableWriterTask - start 3 insert task. 2022-01-10 09:05:07.317 [0-0-10-writer] INFO  ConcurrentTableWriterTask - start 4 insert task. 2022-01-10 09:05:07.356 [0-0-10-writer] INFO  ConcurrentTableWriterTask - start 5 insert task. 2022-01-10 09:05:07.366 [0-0-10-writer] INFO  DBUtil - this is ob1_0 jdbc url. 2022-01-10 09:05:07.366 [0-0-10-writer] INFO  DBUtil - this is ob1_0 jdbc url. user=obce-single:obmysql:root :url=jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true 2022-01-10 09:05:07.375 [0-0-10-writer] INFO  ConcurrentTableWriterTask - ConcurrentTableWriter has put all task in queue, queueSize = 0, total = 0, finished = 0 2022-01-10 09:05:07.469 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[10] is successed, used[639]ms 2022-01-10 09:05:07.573 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[9] is successed, used[2127]ms 2022-01-10 09:05:07.574 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] completed it's tasks. 2022-01-10 09:05:11.356 [job-0] INFO  StandAloneJobContainerCommunicator - Total 100000 records, 18788895 bytes | Speed 1.79MB/s, 10000 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 3.153s | All Task WaitReaderTime 0.719s | Percentage 100.00% 2022-01-10 09:05:11.356 [job-0] INFO  AbstractScheduler - Scheduler accomplished all tasks. 2022-01-10 09:05:11.356 [job-0] INFO  JobContainer - DataX Writer.Job [oceanbasev10writer] do post work. 2022-01-10 09:05:11.356 [job-0] INFO  JobContainer - DataX Reader.Job [mysqlreader] do post work. 2022-01-10 09:05:11.356 [job-0] INFO  JobContainer - DataX jobId [0] completed successfully. 2022-01-10 09:05:11.357 [job-0] INFO  HookInvoker - No hook invoked, because base dir not exists or is a file: /usr/local/datax/hook 2022-01-10 09:05:11.358 [job-0] INFO  JobContainer - [total cpu info] => averageCpu | maxDeltaCpu | minDeltaCpu -1.00% | -1.00% | -1.00% [total gc info] => NAME | totalGCCount | maxDeltaGCCount | minDeltaGCCount | totalGCTime | maxDeltaGCTime | minDeltaGCTime     
                 PS MarkSweep | 0 | 0 | 0 | 0.000s | 0.000s | 0.000s             
                 PS Scavenge | 3 | 3 | 3 | 0.237s | 0.237s | 0.237s 2022-01-10 09:05:11.358 [job-0] INFO  JobContainer - PerfTrace not enable! 2022-01-10 09:05:11.359 [job-0] INFO  StandAloneJobContainerCommunicator - Total 100000 records, 18788895 bytes | Speed 1.79MB/s, 10000 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 3.153s | All Task WaitReaderTime 0.719s | Percentage 100.00% 2022-01-10 09:05:11.360 [job-0] INFO  JobContainer - 任务启动时刻                    : 2022-01-10 09:04:59 任务结束时刻                    : 2022-01-10 09:05:11 任务总计耗时                    : 11s
任务平均流量                    : 1.79MB/s
记录写入速度                    : 10000rec/s
读出记录总数                    : 100000 读写失败总数                    : 0
datax 实现 OceanBase 以 csv 格式的导入导出 OceanBase 以 csv 格式导出数据

生成 datax 配置 json 模板 

python2 /usr/local/datax/bin/datax.py -r oceanbasev10reader -w txtfilewriter > /usr/local/datax/job/oceanbase2csv.json

修改 json 文件： 

cat > /usr/local/datax/job/oceanbase2csv.json <<"EOF" { "job": { "setting": { "speed": { "channel": 4 }, "errorLimit": { "record": 0, "percentage": 0.1 }
       }, "content": [ { "reader": { "name": "oceanbasev10reader", "parameter": { "column": ["*"], "connection": [ { "jdbcUrl": ["||_dsc_ob10_dsc_||obce-single:obmysql||_dsc_ob10_dsc_||jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8"], "table": ["sbtest1"] } ], "password": "lhr", "username": "root", "where": "" }
               }, "writer": { "name": "txtfilewriter", "parameter": { "dateFormat": "yyyy-MM-dd hh:mm:ss", "fieldDelimiter": ",", "fileName": "ob2csv", "path": "/tmp/", "writeMode": "truncate" }
               }
           } ] }
}
EOF

执行任务： 

python2 /usr/local/datax/bin/datax.py  /usr/local/datax/job/oceanbase2csv.json

查看结果： 

[admin@0209306f10bf tmp]$ ll
total 19M
drwxr-xr-x 2 admin admin 4.0K Jan 10 18:07 hsperfdata_admin
drwxr-xr-x 2 root  root 4.0K Jan 10 16:22 hsperfdata_root -rw-rw-r-- 1 admin admin  19M Jan 10 18:07 ob2csv__c0bee24e_73cc_4aeb_904d_f337836e95f1 [admin@0209306f10bf tmp]$ more ob2csv__c0bee24e_73cc_4aeb_904d_f337836e95f1 | wc -l 100000 [admin@0209306f10bf tmp]$ head -n 2 ob2csv__c0bee24e_73cc_4aeb_904d_f337836e95f1 1,50147,68487932199-96439406143-93774651418-41631865787-96406072701-20604855487-25459966574-28203206787-41238978918-19503783441,22195207048-70116052123-74140395089-76317954521-98694025897 2,50248,13241531885-45658403807-79170748828-69419634012-13605813761-77983377181-01582588137-21344716829-87370944992-02457486289,28733802923-10548894641-11867531929-71265603657-36546888392 [admin@0209306f10bf tmp]$
OceanBase 以 csv 格式导入数据

生成 datax 配置 json 模板 

python2 /usr/local/datax/bin/datax.py -r oceanbasev10reader -w txtfilewriter > /usr/local/datax/job/csv2oceanbase.json

修改json文件： 

cat > /usr/local/datax/job/csv2oceanbase.json <<"EOF" { "job": { "setting": { "speed": { "channel": 4 }, "errorLimit": { "record": 0, "percentage": 0.1 }
       }, "content": [ { "reader": { "name": "txtfilereader", "parameter": { "path": ["/tmp/ob/"], "fileName": "ob2csv", "encoding": "UTF-8", "column": ["*"], "dateFormat": "yyyy-MM-dd hh:mm:ss" , "nullFormat": "\\N" , "fieldDelimiter": "," }
               }, "writer": { "name": "oceanbasev10writer", "parameter": { "obWriteMode": "insert", "column": ["*"], "preSql": [ "truncate table sbtest11" ], "connection": [ { "jdbcUrl": "||_dsc_ob10_dsc_||obce-single:obmysql||_dsc_ob10_dsc_||jdbc:oceanbase://127.0.0.1:2883/sbtest?useUnicode=true&characterEncoding=utf-8", "table": [ "sbtest11" ] } ], "username": "root", "password": "lhr", "writerThreadCount": 10, "batchSize": 100, "memstoreThreshold": "0.9" }
               }
           } ] }
}
EOF

执行导入操作： 

MySQL [sbtest]> create table sbtest11 like sbtest1; Query OK, 0 rows affected (0.58 sec) MySQL [sbtest]> desc sbtest11; +-------+-----------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +-------+-----------+------+-----+---------+----------------+ | id | int(11) | NO | PRI | NULL | auto_increment | | k | int(11) | NO | MUL | 0 | | | c | char(120) | NO | | | | | pad | char(60) | NO | | | | +-------+-----------+------+-----+---------+----------------+ 4 rows in set (0.06 sec) MySQL [sbtest]> select count(*) from sbtest11; +----------+ | count(*) | +----------+ | 0 | +----------+ 1 row in set (0.05 sec) -- 导入 python2 /usr/local/datax/bin/datax.py /usr/local/datax/job/csv2oceanbase.json

结果：

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-vsN6uewW-1641900939604)(https://cdn.jsdelivr.net/gh/lhrbest/pic/img_2022/202201101817305.png)]

OB数据库查询： 

MySQL [sbtest]> select count(*) from sbtest11; +----------+ | count(*) | +----------+ | 100000 | +----------+ 1 row in set (0.32 sec)
  • 注意：

Path：需要给定文件的路径，该路径下不要有其他文件，执行会报错。

Canal 实现 mysql 与 OceanBase 的增量同步

使用 canal 配置 MySQL 的增量同步到 OceanBase 的 MySQL 租户。

参考：https://www.modb.pro/db/222647

https://open.oceanbase.com/blog/10900160

mysql源端开启二进制日志 
[mysqld]
log-bin=mysql-bin # 开启 binlog
binlog-format=ROW # 选择 ROW 模式
server_id=1 # 配置 MySQL replaction 需要定义，不要和 canal 的 slaveId 重复
部署 canal admin

参考：https://www.xmmup.com/alishujutongbugongjuotterhecanaljianjie.html 

wget https://github.com/alibaba/canal/releases/download/canal-1.1.5/canal.admin-1.1.5.tar.gz tar -zxvf canal.admin-1.1.5.tar.gz -C /usr/local/canal-admin -- 修改配置文件： cd /usr/local/canal-admin/conf
vim application.yml

修改元数据库信息： 

[admin@0209306f10bf conf]$ cat application.yml
server:
  port: 8089 spring:
  jackson: date-format: yyyy-MM-dd HH:mm:ss time-zone: GMT+8 spring.datasource:
  address: 172.17.0.3:3306 database: canal_manager
  username: root
  password: lhr
  driver-class-name: com.mysql.jdbc.Driver
  url: jdbc:mysql://${spring.datasource.address}/${spring.datasource.database}?useUnicode=true&characterEncoding=UTF-8&useSSL=false hikari:
    maximum-pool-size: 30 minimum-idle: 1 canal:
  adminUser: admin
  adminPasswd: admin

初始化元数据： 

mysql -uroot -P3306 -plhr -h 172.17.0.3 source conf/canal_manager.sql -- 查看表 MySQL [sbtest]> use canal_manager; Database changed
MySQL [canal_manager]> show tables; +-------------------------+ | Tables_in_canal_manager | +-------------------------+ | canal_adapter_config | | canal_cluster | | canal_config | | canal_instance_config | | canal_node_server | | canal_user | +-------------------------+ 6 rows in set (0.05 sec)
启动canal 
cd /usr/local/canal-admin/bin/ && ./startup.sh

过程： 

[admin@0209306f10bf local]$  cd /usr/local/canal-admin/bin/ && ./startup.sh
cd to /usr/local/canal-admin/bin for workaround relative path
CLASSPATH :/usr/local/canal-admin/bin/../conf:/usr/local/canal-admin/bin/../lib/zookeeper-3.4.5.jar:/usr/local/canal-admin/bin/../lib/zkclient-0.10.jar:/usr/local/canal-admin/bin/../lib/xmlunit-core-2.5.1.jar:/usr/local/canal-admin/bin/../lib/validation-api-2.0.1.Final.jar:/usr/local/canal-admin/bin/../lib/tomcat-embed-websocket-8.5.29.jar:/usr/local/canal-admin/bin/../lib/tomcat-embed-el-8.5.29.jar:/usr/local/canal-admin/bin/../lib/tomcat-embed-core-8.5.29.jar:/usr/local/canal-admin/bin/../lib/spring-webmvc-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-web-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-tx-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-orm-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-jdbc-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-jcl-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-expression-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-core-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-context-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-test-autoconfigure-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-test-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-web-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-tomcat-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-test-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-logging-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-json-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-jdbc-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-autoconfigure-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-beans-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-aop-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/snakeyaml-1.19.jar:/usr/local/canal-admin/bin/../lib/slf4j-api-1.7.25.jar:/usr/local/canal-admin/bin/../lib/protobuf-java-3.6.1.jar:/usr/local/canal-admin/bin/../lib/persistence-api-2.2.2.jar:/usr/local/canal-admin/bin/../lib/objenesis-2.6.jar:/usr/local/canal-admin/bin/../lib/netty-all-4.1.23.Final.jar:/usr/local/canal-admin/bin/../lib/netty-3.2.2.Final.jar:/usr/local/canal-admin/bin/../lib/mysql-connector-java-5.1.48.jar:/usr/local/canal-admin/bin/../lib/mockito-core-2.15.0.jar:/usr/local/canal-admin/bin/../lib/logback-core-1.2.3.jar:/usr/local/canal-admin/bin/../lib/logback-classic-1.2.3.jar:/usr/local/canal-admin/bin/../lib/log4j-to-slf4j-2.10.0.jar:/usr/local/canal-admin/bin/../lib/log4j-api-2.10.0.jar:/usr/local/canal-admin/bin/../lib/jul-to-slf4j-1.7.25.jar:/usr/local/canal-admin/bin/../lib/jsr305-3.0.2.jar:/usr/local/canal-admin/bin/../lib/jsonassert-1.5.0.jar:/usr/local/canal-admin/bin/../lib/json-smart-2.3.jar:/usr/local/canal-admin/bin/../lib/json-path-2.4.0.jar:/usr/local/canal-admin/bin/../lib/jcl-over-slf4j-1.7.25.jar:/usr/local/canal-admin/bin/../lib/jboss-logging-3.3.2.Final.jar:/usr/local/canal-admin/bin/../lib/javax.annotation-api-1.3.2.jar:/usr/local/canal-admin/bin/../lib/jackson-module-parameter-names-2.9.5.jar:/usr/local/canal-admin/bin/../lib/jackson-datatype-jsr310-2.9.5.jar:/usr/local/canal-admin/bin/../lib/jackson-datatype-jdk8-2.9.5.jar:/usr/local/canal-admin/bin/../lib/jackson-databind-2.9.5.jar:/usr/local/canal-admin/bin/../lib/jackson-core-2.9.5.jar:/usr/local/canal-admin/bin/../lib/jackson-annotations-2.9.0.jar:/usr/local/canal-admin/bin/../lib/j2objc-annotations-1.1.jar:/usr/local/canal-admin/bin/../lib/hibernate-validator-6.0.9.Final.jar:/usr/local/canal-admin/bin/../lib/hamcrest-library-1.3.jar:/usr/local/canal-admin/bin/../lib/hamcrest-core-1.3.jar:/usr/local/canal-admin/bin/../lib/guava-22.0.jar:/usr/local/canal-admin/bin/../lib/fastjson-1.2.58.sec06.jar:/usr/local/canal-admin/bin/../lib/error_prone_annotations-2.0.18.jar:/usr/local/canal-admin/bin/../lib/ebean-types-1.3.jar:/usr/local/canal-admin/bin/../lib/ebean-migration-11.16.2.jar:/usr/local/canal-admin/bin/../lib/ebean-datasource-api-4.5.jar:/usr/local/canal-admin/bin/../lib/ebean-datasource-4.5.2.jar:/usr/local/canal-admin/bin/../lib/ebean-annotation-4.11.jar:/usr/local/canal-admin/bin/../lib/ebean-11.41.1.jar:/usr/local/canal-admin/bin/../lib/commons-logging-1.1.1.jar:/usr/local/canal-admin/bin/../lib/commons-lang-2.6.jar:/usr/local/canal-admin/bin/../lib/commons-io-2.4.jar:/usr/local/canal-admin/bin/../lib/commons-codec-1.11.jar:/usr/local/canal-admin/bin/../lib/commons-beanutils-1.8.2.jar:/usr/local/canal-admin/bin/../lib/classmate-1.3.4.jar:/usr/local/canal-admin/bin/../lib/canal.protocol-1.1.5.jar:/usr/local/canal-admin/bin/../lib/canal.common-1.1.5.jar:/usr/local/canal-admin/bin/../lib/canal.admin-web-1.1.5.jar:/usr/local/canal-admin/bin/../lib/caffeine-2.6.2.jar:/usr/local/canal-admin/bin/../lib/byte-buddy-agent-1.7.11.jar:/usr/local/canal-admin/bin/../lib/byte-buddy-1.7.11.jar:/usr/local/canal-admin/bin/../lib/avaje-classpath-scanner-api-2.2.jar:/usr/local/canal-admin/bin/../lib/avaje-classpath-scanner-3.1.1.jar:/usr/local/canal-admin/bin/../lib/assertj-core-3.9.1.jar:/usr/local/canal-admin/bin/../lib/asm-5.0.4.jar:/usr/local/canal-admin/bin/../lib/antlr4-runtime-4.7.2.jar:/usr/local/canal-admin/bin/../lib/animal-sniffer-annotations-1.14.jar:/usr/local/canal-admin/bin/../lib/android-json-0.0.20131108.vaadin1.jar:/usr/local/canal-admin/bin/../lib/accessors-smart-1.2.jar:/usr/local/canal-admin/bin/../lib/HikariCP-2.7.8.jar:
cd to /usr/local/canal-admin/bin for continue [admin@0209306f10bf bin]$ cd /usr/local/canal-admin/bin/ [admin@0209306f10bf bin]$ ll
total 20K -rw-rw-r-- 1 admin admin    6 Jan 10 18:46 admin.pid -rwxr-xr-x 1 admin admin 205 Apr 19 2021 restart.sh -rwxr-xr-x 1 admin admin 747 Apr 19 2021 startup.bat -rwxr-xr-x 1 admin admin 2.0K Apr 19 2021 startup.sh -rwxr-xr-x 1 admin admin 1.4K Apr 19 2021 stop.sh [admin@0209306f10bf bin]$ netstat -tulnp | grep 8089 tcp 0 0 0.0.0.0:8089 0.0.0.0:* LISTEN 28703/java [admin@0209306f10bf bin]$ ps -ef|grep java
admin 28703 0 66 18:46 pts/1 00:01:01 /usr/bin/java -server -Xms1024m -Xmx1024m -XX:+UseG1GC -XX:MaxGCPauseMillis=250 -XX:+UseGCOverheadLimit -XX:+ExplicitGCInvokesConcurrent -XX:+PrintAdaptiveSizePolicy -XX:+PrintTenuringDistribution -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dfile.encoding=UTF-8 -DappName=canal-admin -classpath .:/usr/local/canal-admin/bin/../conf:/usr/local/canal-admin/bin/../lib/zookeeper-3.4.5.jar:/usr/local/canal-admin/bin/../lib/zkclient-0.10.jar:/usr/local/canal-admin/bin/../lib/xmlunit-core-2.5.1.jar:/usr/local/canal-admin/bin/../lib/validation-api-2.0.1.Final.jar:/usr/local/canal-admin/bin/../lib/tomcat-embed-websocket-8.5.29.jar:/usr/local/canal-admin/bin/../lib/tomcat-embed-el-8.5.29.jar:/usr/local/canal-admin/bin/../lib/tomcat-embed-core-8.5.29.jar:/usr/local/canal-admin/bin/../lib/spring-webmvc-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-web-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-tx-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-orm-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-jdbc-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-jcl-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-expression-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-core-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-context-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-test-autoconfigure-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-test-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-web-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-tomcat-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-test-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-logging-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-json-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-jdbc-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-autoconfigure-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-beans-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-aop-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/snakeyaml-1.19.jar:/usr/local/canal-admin/bin/../lib/slf4j-api-1.7.25.jar:/usr/local/canal-admin/bin/../lib/protobuf-java-3.6.1.jar:/usr/local/canal-admin/bin/../lib/persistence-api-2.2.2.jar:/usr/local/canal-admin/bin/../lib/objenesis-2.6.jar:/usr/local/canal-admin/bin/../lib/netty-all-4.1.23.Final.jar:/usr/local/canal-admin/bin/../lib/netty-3.2.2.Final.jar:/usr/local/canal-admin/bin/../lib/mysql-connector-java-5.1.48.jar:/usr/local/canal-admin/bin/../lib/mockito-core-2.15.0.jar:/usr/local/canal-admin/bin/../lib/logback-core-1.2.3.jar:/usr/local/canal-admin/bin/../lib/logback-classic-1.2.3.jar:/usr/local/canal-admin/bin/../lib/log4j-to-slf4j-2.10.0.jar:/usr/local/canal-admin/bin/../lib/log4j-api-2.10.0.jar:/usr/local/canal-admin/bin/../lib/jul-to-slf4j-1.7.25.jar:/usr/local/canal-admin/bin/../lib/jsr305-3.0.2.jar:/usr/local/canal-admin/bin/../lib/jsonassert-1.5.0.jar:/usr/local/canal-admin/bin/../lib/json-smart-2.3.jar:/usr/local/canal-admin/bin/../lib/json-path-2.4.0.jar:/usr/local/canal-admin/bin/../lib/jcl-over-slf4j-1.7.25.jar:/usr/local/canal-admin/bin/../lib/jboss-logging-3.3.2.Final.jar:/usr/local/canal-admin/bin/../lib/javax.annotation-api-1.3.2.jar:/usr/local/canal-admin/bin/../lib/jackson-module-parameter-names-2.9.5.jar:/usr/local/canal-admin/bin/../lib/jackson-datatype-jsr310-2.9.5.jar:/usr/local/canal-admin/bin/../lib/jackson-datatype-jdk8-2.9.5.jar:/usr/local/canal-admin/bin/../lib/jackson-databind-2.9.5.jar:/usr/local/canal-admin/bin/../lib/jackson-core-2.9.5.jar:/usr/local/canal-admin/bin/../lib/jackson-annotations-2.9.0.jar:/usr/local/canal-admin/bin/../lib/j2objc-annotations-1.1.jar:/usr/local/canal-admin/bin/../lib/hibernate-validator-6.0.9.Final.jar:/usr/local/canal-admin/bin/../lib/hamcrest-library-1.3.jar:/usr/local/canal-admin/bin/../lib/hamcrest-core-1.3.jar:/usr/local/canal-admin/bin/../lib/guava-22.0.jar:/usr/local/canal-admin/bin/../lib/fastjson-1.2.58.sec06.jar:/usr/local/canal-admin/bin/../lib/error_prone_annotations-2.0.18.jar:/usr/local/canal-admin/bin/../lib/ebean-types-1.3.jar:/usr/local/canal-admin/bin/../lib/ebean-migration-11.16.2.jar:/usr/local/canal-admin/bin/../lib/ebean-datasource-api-4.5.jar:/usr/local/canal-admin/bin/../lib/ebean-datasource-4.5.2.jar:/usr/local/canal-admin/bin/../lib/ebean-annotation-4.11.jar:/usr/local/canal-admin/bin/../lib/ebean-11.41.1.jar:/usr/local/canal-admin/bin/../lib/commons-logging-1.1.1.jar:/usr/local/canal-admin/bin/../lib/commons-lang-2.6.jar:/usr/local/canal-admin/bin/../lib/commons-io-2.4.jar:/usr/local/canal-admin/bin/../lib/commons-codec-1.11.jar:/usr/local/canal-admin/bin/../lib/commons-beanutils-1.8.2.jar:/usr/local/canal-admin/bin/../lib/classmate-1.3.4.jar:/usr/local/canal-admin/bin/../lib/canal.protocol-1.1.5.jar:/usr/local/canal-admin/bin/../lib/canal.common-1.1.5.jar:/usr/local/canal-admin/bin/../lib/canal.admin-web-1.1.5.jar:/usr/local/canal-admin/bin/../lib/caffeine-2.6.2.jar:/usr/local/canal-admin/bin/../lib/byte-buddy-agent-1.7.11.jar:/usr/local/canal-admin/bin/../lib/byte-buddy-1.7.11.jar:/usr/local/canal-admin/bin/../lib/avaje-classpath-scanner-api-2.2.jar:/usr/local/canal-admin/bin/../lib/avaje-classpath-scanner-3.1.1.jar:/usr/local/canal-admin/bin/../lib/assertj-core-3.9.1.jar:/usr/local/canal-admin/bin/../lib/asm-5.0.4.jar:/usr/local/canal-admin/bin/../lib/antlr4-runtime-4.7.2.jar:/usr/local/canal-admin/bin/../lib/animal-sniffer-annotations-1.14.jar:/usr/local/canal-admin/bin/../lib/android-json-0.0.20131108.vaadin1.jar:/usr/local/canal-admin/bin/../lib/accessors-smart-1.2.jar:/usr/local/canal-admin/bin/../lib/HikariCP-2.7.8.jar: com.alibaba.otter.canal.admin.CanalAdminApplication
admin 29139 23 0 18:47 pts/1 00:00:00 grep --color=auto java

查看 canal-admin 的 web 服务:

http://172.17.0.2:8089 admin/123456

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-gg02YP7e-1641900939604)(https://cdn.jsdelivr.net/gh/lhrbest/pic/img_2022/202201101852511.png)]

这里，需要在图形界面修改密码为admin123 ,然后在mysql中查询： 

MySQL [canal_manager]> select * from canal_user; +----+----------+------------------------------------------+---------------+-------+--------------+--------+---------------------+ | id | username | password | name | roles | introduction | avatar | creation_date | +----+----------+------------------------------------------+---------------+-------+--------------+--------+---------------------+ | 1 | admin | 01a6717b58ff5c7eafff6cb7c96f7428ea65fe4c | Canal Manager | admin | NULL | NULL | 2022-01-10 19:25:50 | +----+----------+------------------------------------------+---------------+-------+--------------+--------+---------------------+ 1 row in set (0.23 sec)
部署canal deployer

下载安装canal deployer并解压 

wget https://github.com/alibaba/canal/releases/download/canal-1.1.5/canal.deployer-1.1.5.tar.gz
tar -zxvf canal.deployer-1.1.5.tar.gz -C /usr/local/canal-deployer

修改配置文件

canal Admin 部署 server 和 instance，配置文件 conf/canal_local.properties 替换 conf/canal.properties。需要修改 conf/canal.properties 里的 manager 地址，其他参数值可以保持默认。 

cd /usr/local/canal-deployer/conf
mv canal_local.properties canal.properties && vim canal.properties [admin@0209306f10bf conf]$ cat canal.properties # register ip canal.register.ip = # canal admin config canal.admin.manager = 127.0.0.1:8089 canal.admin.port = 11110 canal.admin.user = admin
canal.admin.passwd = 01a6717b58ff5c7eafff6cb7c96f7428ea65fe4c # admin auto register canal.admin.register.auto = true canal.admin.register.cluster = canal.admin.register.name =

这里是本机部署，需要修改canal.admin.passwd即可。

启动 canal-deployer 服务 
cd /usr/local/canal-deployer/bin/ && ./startup.sh

查看： 

[admin@0209306f10bf ~]$ ps -ef|grep java
admin 3097 0 6 19:27 pts/1 00:00:05 /usr/bin/java -server -Xms2048m -Xmx3072m -Xmn1024m -XX:SurvivorRatio=2 -XX:PermSize=96m -XX:MaxPermSize=256m -Xss256k -XX:-UseAdaptiveSizePolicy -XX:MaxTenuringThreshold=15 -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+UseCMSCompactAtFullCollection -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dfile.encoding=UTF-8 -DappName=otter-canal -Dlogback.configurationFile=/usr/local/canal-deployer/bin/../conf/logback.xml -Dcanal.conf=/usr/local/canal-deployer/bin/../conf/canal.properties -classpath .:/usr/local/canal-deployer/bin/../conf:/usr/local/canal-deployer/bin/../lib/zookeeper-3.4.5.jar:/usr/local/canal-deployer/bin/../lib/zkclient-0.10.jar:/usr/local/canal-deployer/bin/../lib/spring-tx-5.0.5.RELEASE.jar:/usr/local/canal-deployer/bin/../lib/spring-orm-5.0.5.RELEASE.jar:/usr/local/canal-deployer/bin/../lib/spring-jdbc-5.0.5.RELEASE.jar:/usr/local/canal-deployer/bin/../lib/spring-jcl-5.0.5.RELEASE.jar:/usr/local/canal-deployer/bin/../lib/spring-expression-5.0.5.RELEASE.jar:/usr/local/canal-deployer/bin/../lib/spring-core-5.0.5.RELEASE.jar:/usr/local/canal-deployer/bin/../lib/spring-context-5.0.5.RELEASE.jar:/usr/local/canal-deployer/bin/../lib/spring-beans-5.0.5.RELEASE.jar:/usr/local/canal-deployer/bin/../lib/spring-aop-5.0.5.RELEASE.jar:/usr/local/canal-deployer/bin/../lib/slf4j-api-1.7.12.jar:/usr/local/canal-deployer/bin/../lib/simpleclient_pushgateway-0.4.0.jar:/usr/local/canal-deployer/bin/../lib/simpleclient_httpserver-0.4.0.jar:/usr/local/canal-deployer/bin/../lib/simpleclient_hotspot-0.4.0.jar:/usr/local/canal-deployer/bin/../lib/simpleclient_common-0.4.0.jar:/usr/local/canal-deployer/bin/../lib/simpleclient-0.4.0.jar:/usr/local/canal-deployer/bin/../lib/protobuf-java-3.6.1.jar:/usr/local/canal-deployer/bin/../lib/oro-2.0.8.jar:/usr/local/canal-deployer/bin/../lib/netty-all-4.1.6.Final.jar:/usr/local/canal-deployer/bin/../lib/netty-3.2.2.Final.jar:/usr/local/canal-deployer/bin/../lib/mysql-connector-java-5.1.48.jar:/usr/local/canal-deployer/bin/../lib/mybatis-spring-2.0.4.jar:/usr/local/canal-deployer/bin/../lib/mybatis-3.5.4.jar:/usr/local/canal-deployer/bin/../lib/logback-core-1.1.3.jar:/usr/local/canal-deployer/bin/../lib/logback-classic-1.1.3.jar:/usr/local/canal-deployer/bin/../lib/jsr305-3.0.2.jar:/usr/local/canal-deployer/bin/../lib/joda-time-2.9.4.jar:/usr/local/canal-deployer/bin/../lib/jctools-core-2.1.2.jar:/usr/local/canal-deployer/bin/../lib/jcl-over-slf4j-1.7.12.jar:/usr/local/canal-deployer/bin/../lib/javax.annotation-api-1.3.2.jar:/usr/local/canal-deployer/bin/../lib/j2objc-annotations-1.1.jar:/usr/local/canal-deployer/bin/../lib/httpcore-4.4.3.jar:/usr/local/canal-deployer/bin/../lib/httpclient-4.5.1.jar:/usr/local/canal-deployer/bin/../lib/h2-1.4.196.jar:/usr/local/canal-deployer/bin/../lib/guava-22.0.jar:/usr/local/canal-deployer/bin/../lib/fastjson-1.2.58.sec06.jar:/usr/local/canal-deployer/bin/../lib/error_prone_annotations-2.0.18.jar:/usr/local/canal-deployer/bin/../lib/druid-1.2.6.jar:/usr/local/canal-deployer/bin/../lib/disruptor-3.4.2.jar:/usr/local/canal-deployer/bin/../lib/connector.core-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/commons-logging-1.2.jar:/usr/local/canal-deployer/bin/../lib/commons-lang3-3.7.jar:/usr/local/canal-deployer/bin/../lib/commons-lang-2.6.jar:/usr/local/canal-deployer/bin/../lib/commons-io-2.4.jar:/usr/local/canal-deployer/bin/../lib/commons-compress-1.9.jar:/usr/local/canal-deployer/bin/../lib/commons-codec-1.9.jar:/usr/local/canal-deployer/bin/../lib/commons-beanutils-1.8.2.jar:/usr/local/canal-deployer/bin/../lib/canal.store-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.sink-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.server-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.protocol-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.prometheus-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.parse.driver-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.parse.dbsync-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.parse-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.meta-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.instance.spring-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.instance.manager-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.instance.core-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.filter-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.deployer-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.common-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/aviator-2.2.1.jar:/usr/local/canal-deployer/bin/../lib/animal-sniffer-annotations-1.14.jar: com.alibaba.otter.canal.deployer.CanalLauncher
admin 3327 23 0 19:28 pts/1 00:00:00 grep --color=auto java admin 31216 0 4 19:01 pts/1 00:01:11 /usr/bin/java -server -Xms2048m -Xmx3072m -XX:+UseG1GC -XX:MaxGCPauseMillis=250 -XX:+UseGCOverheadLimit -XX:+ExplicitGCInvokesConcurrent -XX:+PrintAdaptiveSizePolicy -XX:+PrintTenuringDistribution -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dfile.encoding=UTF-8 -DappName=canal-admin -classpath .:/usr/local/canal-admin/bin/../conf:/usr/local/canal-admin/bin/../lib/zookeeper-3.4.5.jar:/usr/local/canal-admin/bin/../lib/zkclient-0.10.jar:/usr/local/canal-admin/bin/../lib/xmlunit-core-2.5.1.jar:/usr/local/canal-admin/bin/../lib/validation-api-2.0.1.Final.jar:/usr/local/canal-admin/bin/../lib/tomcat-embed-websocket-8.5.29.jar:/usr/local/canal-admin/bin/../lib/tomcat-embed-el-8.5.29.jar:/usr/local/canal-admin/bin/../lib/tomcat-embed-core-8.5.29.jar:/usr/local/canal-admin/bin/../lib/spring-webmvc-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-web-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-tx-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-orm-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-jdbc-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-jcl-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-expression-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-core-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-context-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-test-autoconfigure-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-test-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-web-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-tomcat-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-test-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-logging-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-json-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-jdbc-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-autoconfigure-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-beans-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-aop-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/snakeyaml-1.19.jar:/usr/local/canal-admin/bin/../lib/slf4j-api-1.7.25.jar:/usr/local/canal-admin/bin/../lib/protobuf-java-3.6.1.jar:/usr/local/canal-admin/bin/../lib/persistence-api-2.2.2.jar:/usr/local/canal-admin/bin/../lib/objenesis-2.6.jar:/usr/local/canal-admin/bin/../lib/netty-all-4.1.23.Final.jar:/usr/local/canal-admin/bin/../lib/netty-3.2.2.Final.jar:/usr/local/canal-admin/bin/../lib/mysql-connector-java-5.1.48.jar:/usr/local/canal-admin/bin/../lib/mockito-core-2.15.0.jar:/usr/local/canal-admin/bin/../lib/logback-core-1.2.3.jar:/usr/local/canal-admin/bin/../lib/logback-classic-1.2.3.jar:/usr/local/canal-admin/bin/../lib/log4j-to-slf4j-2.10.0.jar:/usr/local/canal-admin/bin/../lib/log4j-api-2.10.0.jar:/usr/local/canal-admin/bin/../lib/jul-to-slf4j-1.7.25.jar:/usr/local/canal-admin/bin/../lib/jsr305-3.0.2.jar:/usr/local/canal-admin/bin/../lib/jsonassert-1.5.0.jar:/usr/local/canal-admin/bin/../lib/json-smart-2.3.jar:/usr/local/canal-admin/bin/../lib/json-path-2.4.0.jar:/usr/local/canal-admin/bin/../lib/jcl-over-slf4j-1.7.25.jar:/usr/local/canal-admin/bin/../lib/jboss-logging-3.3.2.Final.jar:/usr/local/canal-admin/bin/../lib/javax.annotation-api-1.3.2.jar:/usr/local/canal-admin/bin/../lib/jackson-module-parameter-names-2.9.5.jar:/usr/local/canal-admin/bin/../lib/jackson-datatype-jsr310-2.9.5.jar:/usr/local/canal-admin/bin/../lib/jackson-datatype-jdk8-2.9.5.jar:/usr/local/canal-admin/bin/../lib/jackson-databind-2.9.5.jar:/usr/local/canal-admin/bin/../lib/jackson-core-2.9.5.jar:/usr/local/canal-admin/bin/../lib/jackson-annotations-2.9.0.jar:/usr/local/canal-admin/bin/../lib/j2objc-annotations-1.1.jar:/usr/local/canal-admin/bin/../lib/hibernate-validator-6.0.9.Final.jar:/usr/local/canal-admin/bin/../lib/hamcrest-library-1.3.jar:/usr/local/canal-admin/bin/../lib/hamcrest-core-1.3.jar:/usr/local/canal-admin/bin/../lib/guava-22.0.jar:/usr/local/canal-admin/bin/../lib/fastjson-1.2.58.sec06.jar:/usr/local/canal-admin/bin/../lib/error_prone_annotations-2.0.18.jar:/usr/local/canal-admin/bin/../lib/ebean-types-1.3.jar:/usr/local/canal-admin/bin/../lib/ebean-migration-11.16.2.jar:/usr/local/canal-admin/bin/../lib/ebean-datasource-api-4.5.jar:/usr/local/canal-admin/bin/../lib/ebean-datasource-4.5.2.jar:/usr/local/canal-admin/bin/../lib/ebean-annotation-4.11.jar:/usr/local/canal-admin/bin/../lib/ebean-11.41.1.jar:/usr/local/canal-admin/bin/../lib/commons-logging-1.1.1.jar:/usr/local/canal-admin/bin/../lib/commons-lang-2.6.jar:/usr/local/canal-admin/bin/../lib/commons-io-2.4.jar:/usr/local/canal-admin/bin/../lib/commons-codec-1.11.jar:/usr/local/canal-admin/bin/../lib/commons-beanutils-1.8.2.jar:/usr/local/canal-admin/bin/../lib/classmate-1.3.4.jar:/usr/local/canal-admin/bin/../lib/canal.protocol-1.1.5.jar:/usr/local/canal-admin/bin/../lib/canal.common-1.1.5.jar:/usr/local/canal-admin/bin/../lib/canal.admin-web-1.1.5.jar:/usr/local/canal-admin/bin/../lib/caffeine-2.6.2.jar:/usr/local/canal-admin/bin/../lib/byte-buddy-agent-1.7.11.jar:/usr/local/canal-admin/bin/../lib/byte-buddy-1.7.11.jar:/usr/local/canal-admin/bin/../lib/avaje-classpath-scanner-api-2.2.jar:/usr/local/canal-admin/bin/../lib/avaje-classpath-scanner-3.1.1.jar:/usr/local/canal-admin/bin/../lib/assertj-core-3.9.1.jar:/usr/local/canal-admin/bin/../lib/asm-5.0.4.jar:/usr/local/canal-admin/bin/../lib/antlr4-runtime-4.7.2.jar:/usr/local/canal-admin/bin/../lib/animal-sniffer-annotations-1.14.jar:/usr/local/canal-admin/bin/../lib/android-json-0.0.20131108.vaadin1.jar:/usr/local/canal-admin/bin/../lib/accessors-smart-1.2.jar:/usr/local/canal-admin/bin/../lib/HikariCP-2.7.8.jar: com.alibaba.otter.canal.admin.CanalAdminApplication [admin@0209306f10bf ~]$ netstat -tulnp | grep 11110 tcp 0 0 0.0.0.0:11110 0.0.0.0:* LISTEN 3097/java [admin@0209306f10bf ~]$ netstat -tulnp | grep 8089 tcp 0 0 0.0.0.0:8089 0.0.0.0:* LISTEN 31216/java
通过 canal-admin 的 web 服务配置 canal server 和 canal instance

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-oxdlctEQ-1641900939606)(https://cdn.jsdelivr.net/gh/lhrbest/pic/img_2022/202201101931971.png)]

创建 canal instance 并启动 
#################################################
## mysql serverId , v1.0.26+ will autoGen
canal.instance.mysql.slaveId=57363418

# enable gtid use true/false
canal.instance.gtidon=false

# position info
canal.instance.master.address=172.17.0.3:3306
canal.instance.master.journal.name=
canal.instance.master.position=
canal.instance.master.timestamp=
canal.instance.master.gtid=

# rds oss binlog
canal.instance.rds.accesskey=
canal.instance.rds.secretkey=
canal.instance.rds.instanceId=

# table meta tsdb info
canal.instance.tsdb.enable=true
#canal.instance.tsdb.url=jdbc:mysql://127.0.0.1:3306/canal_tsdb
#canal.instance.tsdb.dbUsername=canal
#canal.instance.tsdb.dbPassword=canal

#canal.instance.standby.address =
#canal.instance.standby.journal.name =
#canal.instance.standby.position =
#canal.instance.standby.timestamp =
#canal.instance.standby.gtid=

# username/password
canal.instance.dbUsername=root
canal.instance.dbPassword=lhr
canal.instance.connectionCharset = UTF-8
# enable druid Decrypt database password
canal.instance.enableDruid=false
#canal.instance.pwdPublicKey=MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBALK4BUxdDltRRE5/zXpVEVPUgunvscYFtEip3pmLlhrWpacX7y7GCMo2/JM6LeHmiiNdH1FWgGCpUfircSwlWKUCAwEAAQ==

# table regex
canal.instance.filter.regex=.*\\..*
# table black regex
canal.instance.filter.black.regex=
# table field filter(format: schema1.tableName1:field1/field2,schema2.tableName2:field1/field2)
#canal.instance.filter.field=test1.t_product:id/subject/keywords,test2.t_company:id/name/contact/ch
# table field black filter(format: schema1.tableName1:field1/field2,schema2.tableName2:field1/field2)
#canal.instance.filter.black.field=test1.t_product:subject/product_image,test2.t_company:id/name/contact/ch

# mq config
canal.mq.topic=example
# dynamic topic route by schema or table regex
#canal.mq.dynamicTopic=mytest1.user,mytest2\\..*,.*\\..*
canal.mq.partition=0
# hash partition config
#canal.mq.partitionsNum=3
#canal.mq.partitionHash=test.table:id^name,.*\\..*
#################################################
部署 canal adapter 
wget https://github.com/alibaba/canal/releases/download/canal-1.1.5/canal.adapter-1.1.5.tar.gz tar -zxvf canal.adapter-1.1.5.tar.gz -C /usr/local/canal-adapter

cd /usr/local/canal-adapter/conf/ && vim application.yml


cat > /usr/local/canal-adapter/conf/application.yml <<"EOF" server:
 port: 8081 spring:
 jackson: date-format: yyyy-MM-dd HH:mm:ss time-zone: GMT+8 default-property-inclusion: non_null
 
canal.conf: mode: tcp #tcp kafka rocketMQ rabbitMQ flatMessage: true zookeeperHosts:
 syncBatchSize: 1000 retries: 0 timeout:
 accessKey:
 secretKey:
 consumerProperties: # canal tcp consumer canal.tcp.server.host: 127.0.0.1:11111 canal.tcp.zookeeper.hosts:
   canal.tcp.batch.size: 500 canal.tcp.username: admin
   canal.tcp.password: 4ACFE3202A5FF5CF467898FC58AAB1D615029441
 canalAdapters: - instance: canal1 # canal instance Name or mq topic name groups: - groupId: group1
     outerAdapters: - name: logger - name: rdb key: oceanbase1
       properties:
         jdbc.driverClassName: com.mysql.jdbc.Driver
         jdbc.url: jdbc:mysql://172.17.0.2:2883/sbtest?useUnicode=true jdbc.username: root@sys#obce-single jdbc.password: lhr
EOF

修改RDB 映射文件：conf/rdb/mytest_user.yml 

cat > /usr/local/canal-adapter/conf/rdb/mytest_user.yml <<"EOF"
dataSourceKey: defaultDS
destination: canal1 
groupId: g1
outerAdapterKey: oceanbase1
concurrent: true
dbMapping:
  mirrorDb: true
  database: sbtest
EOF
  • destination: 指定的是canal instance名称
  • outerAdapterKey: 指定的是application.yml文件中的key
启动 canal adapter 服务 
cd /usr/local/canal-adapter/bin/ && ./startup.sh

进程： 

[admin@0209306f10bf bin]$ ps -ef|grep java
admin 1537 0 99 17:32 pts/3 00:00:53 /usr/bin/java -server -Xms2048m -Xmx3072m -Xmn1024m -XX:SurvivorRatio=2 -XX:PermSize=96m -XX:MaxPermSize=256m -Xss256k -XX:-UseAdaptiveSizePolicy -XX:MaxTenuringThreshold=15 -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+UseCMSCompactAtFullCollection -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dfile.encoding=UTF-8 -DappName=canal-adapter -classpath .:/usr/local/canal-adapter/bin/../conf:/usr/local/canal-adapter/bin/../lib/zookeeper-3.4.5.jar:/usr/local/canal-adapter/bin/../lib/zkclient-0.10.jar:/usr/local/canal-adapter/bin/../lib/validation-api-2.0.1.Final.jar:/usr/local/canal-adapter/bin/../lib/tomcat-embed-websocket-8.5.29.jar:/usr/local/canal-adapter/bin/../lib/tomcat-embed-el-8.5.29.jar:/usr/local/canal-adapter/bin/../lib/tomcat-embed-core-8.5.29.jar:/usr/local/canal-adapter/bin/../lib/spring-webmvc-5.0.5.RELEASE.jar:/usr/local/canal-adapter/bin/../lib/spring-web-5.0.5.RELEASE.jar:/usr/local/canal-adapter/bin/../lib/spring-tx-5.0.5.RELEASE.jar:/usr/local/canal-adapter/bin/../lib/spring-security-crypto-5.0.4.RELEASE.jar:/usr/local/canal-adapter/bin/../lib/spring-orm-5.0.5.RELEASE.jar:/usr/local/canal-adapter/bin/../lib/spring-jdbc-5.0.5.RELEASE.jar:/usr/local/canal-adapter/bin/../lib/spring-jcl-5.0.5.RELEASE.jar:/usr/local/canal-adapter/bin/../lib/spring-expression-5.0.5.RELEASE.jar:/usr/local/canal-adapter/bin/../lib/spring-core-5.0.5.RELEASE.jar:/usr/local/canal-adapter/bin/../lib/spring-context-5.0.5.RELEASE.jar:/usr/local/canal-adapter/bin/../lib/spring-cloud-context-2.0.0.RELEASE.jar:/usr/local/canal-adapter/bin/../lib/spring-boot-starter-web-2.0.1.RELEASE.jar:/usr/local/canal-adapter/bin/../lib/spring-boot-starter-tomcat-2.0.1.RELEASE.jar:/usr/local/canal-adapter/bin/../lib/spring-boot-starter-logging-2.0.1.RELEASE.jar:/usr/local/canal-adapter/bin/../lib/spring-boot-starter-json-2.0.1.RELEASE.jar:/usr/local/canal-adapter/bin/../lib/spring-boot-starter-2.0.1.RELEASE.jar:/usr/local/canal-adapter/bin/../lib/spring-boot-autoconfigure-2.0.1.RELEASE.jar:/usr/local/canal-adapter/bin/../lib/spring-boot-2.0.1.RELEASE.jar:/usr/local/canal-adapter/bin/../lib/spring-beans-5.0.5.RELEASE.jar:/usr/local/canal-adapter/bin/../lib/spring-aop-5.0.5.RELEASE.jar:/usr/local/canal-adapter/bin/../lib/snakeyaml-1.19.jar:/usr/local/canal-adapter/bin/../lib/slf4j-api-1.7.12.jar:/usr/local/canal-adapter/bin/../lib/protobuf-java-3.6.1.jar:/usr/local/canal-adapter/bin/../lib/postgresql-42.1.4.jar:/usr/local/canal-adapter/bin/../lib/oro-2.0.8.jar:/usr/local/canal-adapter/bin/../lib/ojdbc6-11.2.0.3.jar:/usr/local/canal-adapter/bin/../lib/netty-all-4.1.6.Final.jar:/usr/local/canal-adapter/bin/../lib/netty-3.2.2.Final.jar:/usr/local/canal-adapter/bin/../lib/mysql-connector-java-5.1.48.jar:/usr/local/canal-adapter/bin/../lib/mssql-jdbc-7.0.0.jre8.jar:/usr/local/canal-adapter/bin/../lib/logback-core-1.1.3.jar:/usr/local/canal-adapter/bin/../lib/logback-classic-1.1.3.jar:/usr/local/canal-adapter/bin/../lib/log4j-to-slf4j-2.10.0.jar:/usr/local/canal-adapter/bin/../lib/log4j-api-2.10.0.jar:/usr/local/canal-adapter/bin/../lib/log4j-1.2.17.jar:/usr/local/canal-adapter/bin/../lib/jul-to-slf4j-1.7.25.jar:/usr/local/canal-adapter/bin/../lib/jsr305-3.0.2.jar:/usr/local/canal-adapter/bin/../lib/joda-time-2.9.4.jar:/usr/local/canal-adapter/bin/../lib/jcl-over-slf4j-1.7.12.jar:/usr/local/canal-adapter/bin/../lib/jboss-logging-3.3.2.Final.jar:/usr/local/canal-adapter/bin/../lib/javax.annotation-api-1.3.2.jar:/usr/local/canal-adapter/bin/../lib/jackson-module-parameter-names-2.9.5.jar:/usr/local/canal-adapter/bin/../lib/jackson-datatype-jsr310-2.9.5.jar:/usr/local/canal-adapter/bin/../lib/jackson-datatype-jdk8-2.9.5.jar:/usr/local/canal-adapter/bin/../lib/jackson-databind-2.9.5.jar:/usr/local/canal-adapter/bin/../lib/jackson-core-2.9.5.jar:/usr/local/canal-adapter/bin/../lib/jackson-annotations-2.9.0.jar:/usr/local/canal-adapter/bin/../lib/j2objc-annotations-1.1.jar:/usr/local/canal-adapter/bin/../lib/htrace-core-3.1.0-incubating.jar:/usr/local/canal-adapter/bin/../lib/hibernate-validator-6.0.9.Final.jar:/usr/local/canal-adapter/bin/../lib/hbase-shaded-client-1.1.2.jar:/usr/local/canal-adapter/bin/../lib/guava-22.0.jar:/usr/local/canal-adapter/bin/../lib/findbugs-annotations-1.3.9-1.jar:/usr/local/canal-adapter/bin/../lib/fastjson-1.2.58.sec06.jar:/usr/local/canal-adapter/bin/../lib/error_prone_annotations-2.0.18.jar:/usr/local/canal-adapter/bin/../lib/druid-1.2.6.jar:/usr/local/canal-adapter/bin/../lib/curator-recipes-2.10.0.jar:/usr/local/canal-adapter/bin/../lib/curator-framework-2.10.0.jar:/usr/local/canal-adapter/bin/../lib/curator-client-2.10.0.jar:/usr/local/canal-adapter/bin/../lib/connector.core-1.1.5.jar:/usr/local/canal-adapter/bin/../lib/commons-logging-1.2.jar:/usr/local/canal-adapter/bin/../lib/commons-lang-2.6.jar:/usr/local/canal-adapter/bin/../lib/commons-io-2.4.jar:/usr/local/canal-adapter/bin/../lib/commons-codec-1.9.jar:/usr/local/canal-adapter/bin/../lib/commons-beanutils-1.8.2.jar:/usr/local/canal-adapter/bin/../lib/client-adapter.launcher-1.1.5.jar:/usr/local/canal-adapter/bin/../lib/client-adapter.common-1.1.5.jar:/usr/local/canal-adapter/bin/../lib/classmate-1.3.4.jar:/usr/local/canal-adapter/bin/../lib/canal.protocol-1.1.5.jar:/usr/local/canal-adapter/bin/../lib/canal.common-1.1.5.jar:/usr/local/canal-adapter/bin/../lib/aviator-2.2.1.jar:/usr/local/canal-adapter/bin/../lib/animal-sniffer-annotations-1.14.jar: com.alibaba.otter.canal.adapter.launcher.CanalAdapterApplication
admin 3097 0 0 Jan10 pts/1 00:05:17 /usr/bin/java -server -Xms2048m -Xmx3072m -Xmn1024m -XX:SurvivorRatio=2 -XX:PermSize=96m -XX:MaxPermSize=256m -Xss256k -XX:-UseAdaptiveSizePolicy -XX:MaxTenuringThreshold=15 -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+UseCMSCompactAtFullCollection -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dfile.encoding=UTF-8 -DappName=otter-canal -Dlogback.configurationFile=/usr/local/canal-deployer/bin/../conf/logback.xml -Dcanal.conf=/usr/local/canal-deployer/bin/../conf/canal.properties -classpath .:/usr/local/canal-deployer/bin/../conf:/usr/local/canal-deployer/bin/../lib/zookeeper-3.4.5.jar:/usr/local/canal-deployer/bin/../lib/zkclient-0.10.jar:/usr/local/canal-deployer/bin/../lib/spring-tx-5.0.5.RELEASE.jar:/usr/local/canal-deployer/bin/../lib/spring-orm-5.0.5.RELEASE.jar:/usr/local/canal-deployer/bin/../lib/spring-jdbc-5.0.5.RELEASE.jar:/usr/local/canal-deployer/bin/../lib/spring-jcl-5.0.5.RELEASE.jar:/usr/local/canal-deployer/bin/../lib/spring-expression-5.0.5.RELEASE.jar:/usr/local/canal-deployer/bin/../lib/spring-core-5.0.5.RELEASE.jar:/usr/local/canal-deployer/bin/../lib/spring-context-5.0.5.RELEASE.jar:/usr/local/canal-deployer/bin/../lib/spring-beans-5.0.5.RELEASE.jar:/usr/local/canal-deployer/bin/../lib/spring-aop-5.0.5.RELEASE.jar:/usr/local/canal-deployer/bin/../lib/slf4j-api-1.7.12.jar:/usr/local/canal-deployer/bin/../lib/simpleclient_pushgateway-0.4.0.jar:/usr/local/canal-deployer/bin/../lib/simpleclient_httpserver-0.4.0.jar:/usr/local/canal-deployer/bin/../lib/simpleclient_hotspot-0.4.0.jar:/usr/local/canal-deployer/bin/../lib/simpleclient_common-0.4.0.jar:/usr/local/canal-deployer/bin/../lib/simpleclient-0.4.0.jar:/usr/local/canal-deployer/bin/../lib/protobuf-java-3.6.1.jar:/usr/local/canal-deployer/bin/../lib/oro-2.0.8.jar:/usr/local/canal-deployer/bin/../lib/netty-all-4.1.6.Final.jar:/usr/local/canal-deployer/bin/../lib/netty-3.2.2.Final.jar:/usr/local/canal-deployer/bin/../lib/mysql-connector-java-5.1.48.jar:/usr/local/canal-deployer/bin/../lib/mybatis-spring-2.0.4.jar:/usr/local/canal-deployer/bin/../lib/mybatis-3.5.4.jar:/usr/local/canal-deployer/bin/../lib/logback-core-1.1.3.jar:/usr/local/canal-deployer/bin/../lib/logback-classic-1.1.3.jar:/usr/local/canal-deployer/bin/../lib/jsr305-3.0.2.jar:/usr/local/canal-deployer/bin/../lib/joda-time-2.9.4.jar:/usr/local/canal-deployer/bin/../lib/jctools-core-2.1.2.jar:/usr/local/canal-deployer/bin/../lib/jcl-over-slf4j-1.7.12.jar:/usr/local/canal-deployer/bin/../lib/javax.annotation-api-1.3.2.jar:/usr/local/canal-deployer/bin/../lib/j2objc-annotations-1.1.jar:/usr/local/canal-deployer/bin/../lib/httpcore-4.4.3.jar:/usr/local/canal-deployer/bin/../lib/httpclient-4.5.1.jar:/usr/local/canal-deployer/bin/../lib/h2-1.4.196.jar:/usr/local/canal-deployer/bin/../lib/guava-22.0.jar:/usr/local/canal-deployer/bin/../lib/fastjson-1.2.58.sec06.jar:/usr/local/canal-deployer/bin/../lib/error_prone_annotations-2.0.18.jar:/usr/local/canal-deployer/bin/../lib/druid-1.2.6.jar:/usr/local/canal-deployer/bin/../lib/disruptor-3.4.2.jar:/usr/local/canal-deployer/bin/../lib/connector.core-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/commons-logging-1.2.jar:/usr/local/canal-deployer/bin/../lib/commons-lang3-3.7.jar:/usr/local/canal-deployer/bin/../lib/commons-lang-2.6.jar:/usr/local/canal-deployer/bin/../lib/commons-io-2.4.jar:/usr/local/canal-deployer/bin/../lib/commons-compress-1.9.jar:/usr/local/canal-deployer/bin/../lib/commons-codec-1.9.jar:/usr/local/canal-deployer/bin/../lib/commons-beanutils-1.8.2.jar:/usr/local/canal-deployer/bin/../lib/canal.store-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.sink-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.server-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.protocol-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.prometheus-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.parse.driver-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.parse.dbsync-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.parse-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.meta-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.instance.spring-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.instance.manager-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.instance.core-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.filter-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.deployer-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/canal.common-1.1.5.jar:/usr/local/canal-deployer/bin/../lib/aviator-2.2.1.jar:/usr/local/canal-deployer/bin/../lib/animal-sniffer-annotations-1.14.jar: com.alibaba.otter.canal.deployer.CanalLauncher
admin 31216 0 0 Jan10 pts/1 00:10:19 /usr/bin/java -server -Xms2048m -Xmx3072m -XX:+UseG1GC -XX:MaxGCPauseMillis=250 -XX:+UseGCOverheadLimit -XX:+ExplicitGCInvokesConcurrent -XX:+PrintAdaptiveSizePolicy -XX:+PrintTenuringDistribution -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dfile.encoding=UTF-8 -DappName=canal-admin -classpath .:/usr/local/canal-admin/bin/../conf:/usr/local/canal-admin/bin/../lib/zookeeper-3.4.5.jar:/usr/local/canal-admin/bin/../lib/zkclient-0.10.jar:/usr/local/canal-admin/bin/../lib/xmlunit-core-2.5.1.jar:/usr/local/canal-admin/bin/../lib/validation-api-2.0.1.Final.jar:/usr/local/canal-admin/bin/../lib/tomcat-embed-websocket-8.5.29.jar:/usr/local/canal-admin/bin/../lib/tomcat-embed-el-8.5.29.jar:/usr/local/canal-admin/bin/../lib/tomcat-embed-core-8.5.29.jar:/usr/local/canal-admin/bin/../lib/spring-webmvc-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-web-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-tx-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-orm-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-jdbc-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-jcl-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-expression-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-core-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-context-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-test-autoconfigure-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-test-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-web-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-tomcat-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-test-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-logging-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-json-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-jdbc-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-starter-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-autoconfigure-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-boot-2.0.1.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-beans-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/spring-aop-5.0.5.RELEASE.jar:/usr/local/canal-admin/bin/../lib/snakeyaml-1.19.jar:/usr/local/canal-admin/bin/../lib/slf4j-api-1.7.25.jar:/usr/local/canal-admin/bin/../lib/protobuf-java-3.6.1.jar:/usr/local/canal-admin/bin/../lib/persistence-api-2.2.2.jar:/usr/local/canal-admin/bin/../lib/objenesis-2.6.jar:/usr/local/canal-admin/bin/../lib/netty-all-4.1.23.Final.jar:/usr/local/canal-admin/bin/../lib/netty-3.2.2.Final.jar:/usr/local/canal-admin/bin/../lib/mysql-connector-java-5.1.48.jar:/usr/local/canal-admin/bin/../lib/mockito-core-2.15.0.jar:/usr/local/canal-admin/bin/../lib/logback-core-1.2.3.jar:/usr/local/canal-admin/bin/../lib/logback-classic-1.2.3.jar:/usr/local/canal-admin/bin/../lib/log4j-to-slf4j-2.10.0.jar:/usr/local/canal-admin/bin/../lib/log4j-api-2.10.0.jar:/usr/local/canal-admin/bin/../lib/jul-to-slf4j-1.7.25.jar:/usr/local/canal-admin/bin/../lib/jsr305-3.0.2.jar:/usr/local/canal-admin/bin/../lib/jsonassert-1.5.0.jar:/usr/local/canal-admin/bin/../lib/json-smart-2.3.jar:/usr/local/canal-admin/bin/../lib/json-path-2.4.0.jar:/usr/local/canal-admin/bin/../lib/jcl-over-slf4j-1.7.25.jar:/usr/local/canal-admin/bin/../lib/jboss-logging-3.3.2.Final.jar:/usr/local/canal-admin/bin/../lib/javax.annotation-api-1.3.2.jar:/usr/local/canal-admin/bin/../lib/jackson-module-parameter-names-2.9.5.jar:/usr/local/canal-admin/bin/../lib/jackson-datatype-jsr310-2.9.5.jar:/usr/local/canal-admin/bin/../lib/jackson-datatype-jdk8-2.9.5.jar:/usr/local/canal-admin/bin/../lib/jackson-databind-2.9.5.jar:/usr/local/canal-admin/bin/../lib/jackson-core-2.9.5.jar:/usr/local/canal-admin/bin/../lib/jackson-annotations-2.9.0.jar:/usr/local/canal-admin/bin/../lib/j2objc-annotations-1.1.jar:/usr/local/canal-admin/bin/../lib/hibernate-validator-6.0.9.Final.jar:/usr/local/canal-admin/bin/../lib/hamcrest-library-1.3.jar:/usr/local/canal-admin/bin/../lib/hamcrest-core-1.3.jar:/usr/local/canal-admin/bin/../lib/guava-22.0.jar:/usr/local/canal-admin/bin/../lib/fastjson-1.2.58.sec06.jar:/usr/local/canal-admin/bin/../lib/error_prone_annotations-2.0.18.jar:/usr/local/canal-admin/bin/../lib/ebean-types-1.3.jar:/usr/local/canal-admin/bin/../lib/ebean-migration-11.16.2.jar:/usr/local/canal-admin/bin/../lib/ebean-datasource-api-4.5.jar:/usr/local/canal-admin/bin/../lib/ebean-datasource-4.5.2.jar:/usr/local/canal-admin/bin/../lib/ebean-annotation-4.11.jar:/usr/local/canal-admin/bin/../lib/ebean-11.41.1.jar:/usr/local/canal-admin/bin/../lib/commons-logging-1.1.1.jar:/usr/local/canal-admin/bin/../lib/commons-lang-2.6.jar:/usr/local/canal-admin/bin/../lib/commons-io-2.4.jar:/usr/local/canal-admin/bin/../lib/commons-codec-1.11.jar:/usr/local/canal-admin/bin/../lib/commons-beanutils-1.8.2.jar:/usr/local/canal-admin/bin/../lib/classmate-1.3.4.jar:/usr/local/canal-admin/bin/../lib/canal.protocol-1.1.5.jar:/usr/local/canal-admin/bin/../lib/canal.common-1.1.5.jar:/usr/local/canal-admin/bin/../lib/canal.admin-web-1.1.5.jar:/usr/local/canal-admin/bin/../lib/caffeine-2.6.2.jar:/usr/local/canal-admin/bin/../lib/byte-buddy-agent-1.7.11.jar:/usr/local/canal-admin/bin/../lib/byte-buddy-1.7.11.jar:/usr/local/canal-admin/bin/../lib/avaje-classpath-scanner-api-2.2.jar:/usr/local/canal-admin/bin/../lib/avaje-classpath-scanner-3.1.1.jar:/usr/local/canal-admin/bin/../lib/assertj-core-3.9.1.jar:/usr/local/canal-admin/bin/../lib/asm-5.0.4.jar:/usr/local/canal-admin/bin/../lib/antlr4-runtime-4.7.2.jar:/usr/local/canal-admin/bin/../lib/animal-sniffer-annotations-1.14.jar:/usr/local/canal-admin/bin/../lib/android-json-0.0.20131108.vaadin1.jar:/usr/local/canal-admin/bin/../lib/accessors-smart-1.2.jar:/usr/local/canal-admin/bin/../lib/HikariCP-2.7.8.jar: com.alibaba.otter.canal.admin.CanalAdminApplication [admin@0209306f10bf bin]$ [admin@0209306f10bf bin]$ netstat -tulnp
Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address         State       PID/Program name    
tcp 0 0 0.0.0.0:8081 0.0.0.0:* LISTEN 1537/java           
tcp 0 0 0.0.0.0:8089 0.0.0.0:* LISTEN 31216/java          
tcp 0 0 0.0.0.0:2881 0.0.0.0:* LISTEN 859/observer        
tcp 0 0 0.0.0.0:2882 0.0.0.0:* LISTEN 859/observer        
tcp 0 0 0.0.0.0:2883 0.0.0.0:* LISTEN 1475/obproxy        
tcp 0 0 0.0.0.0:2884 0.0.0.0:* LISTEN 1475/obproxy        
tcp 0 0 0.0.0.0:11110 0.0.0.0:* LISTEN 3097/java           
tcp 0 0 0.0.0.0:11111 0.0.0.0:* LISTEN 3097/java           
tcp 0 0 0.0.0.0:11112 0.0.0.0:* LISTEN 3097/java
验证增量数据的同步情况 
create table t2(id int primary key, name varchar(32)) partition by hash(id);

insert into t2 values(1, 'go'),(2, 'python'),(3, 'C++');
commit;

update t2 set name='C' where id = 3;
commit;

delete from t2 where id = 2;

select * from t2;
MySQL [sbtest]> select * from t2;
+----+------+
| id | name |
+----+------+
|  1 | go   |
|  3 | C    |
+----+------+
2 rows in set (0.05 sec)
检查验证目标端数据同步情况 
MySQL [sbtest]> select * from t2; +----+------+ | id | name | +----+------+ | 1 | go | | 3 | C | +----+------+ 2 rows in set (0.05 sec)
关注
打赏
1688896170
查看更多评论

暂无认证

  • 0浏览

    0关注

    105695博文

    0收益

  • 0浏览

    0点赞

    0打赏

    0留言

私信
关注
热门博文

[ 申请 ]友情链接：

快连 快连vpn 搜外友链 笔趣阁 爱思助手 ClashX教程 绘画宝宝 配音宝宝
立即登录/注册

微信扫码登录

0.3683s